LagoonCon Portland 2024 Recap

As we headed into DrupalCon Portland 2024, our first and foremost task, after getting the most out of DrupalCon, was to host a successful DrupalCon Day-1 event: LagoonCon.

With this event, we wanted to provide a platform for discussions around open source, easy app delivery solutions, Drupal, Kubernetes, and generally making the lives of our fellow developers easier. Our first-class lineup was predestined to deliver on the promises made prior to the event:

• Toby Bellwood (amazee.io, Lagoon Product Lead)
• Christophe Weber (Pronovix, Solutions Architect)
• Bryan Gruneberg (Workshop Oragne, CEO & CTO)
• The Lagoon Team
  

If you couldn’t attend this informative and fun event at the time, Worry not. We have a recap of all the sessions ready for you.

Toby explained what Lagoon does and why it’s excellent for devs and ops. Then, our audience recaps what Lagoon has done in the past 12 months. Here are some highlights:

• Focus on security, with more permissions for platform admins and less reliance on legacy tokens.
• Keycloak updated to v17, and then v21 — required major refactoring
• Legacy (Ruby) Pygmy finally deprecated! pygmystack/pygmy all the way now!
• DDEV Support for Lagoon! Check out the webinar from February
• Introduced additional self-service functionalities
• Built image overhaul, better logs, customization, the start of the Go migration
• Multiple component, dependency, and version updates
• Changes to the internal image build, test, and deployment systems
• Individual environment services → gives us a good building platform for “Insights.”
• Documented upgrades and deprecations

So, what’s coming next for Lagoon? Well, there are no idle hands on the Lagoon team:

• Lagoon Retention periods: Using Lagoon to define how long it takes to keep backups, stale images, etc. (represents cost savings and security improvements).
• Lagoon Resource requests: Allowing a Lagoon administrator to set specific scale/redundancy parameters for projects.
• Lagoon Labeling standards: Providing stable, reliable Kubernetes labels to allow Ops teams to build functionality on top of Lagoon (metrics, logging, etc.)
• User Interface v2: Work on the next iteration of the Lagoon UI.

Follow the Lagoon blog, social media, and Discord to be the first to know when new features are released!

Christoph Weber, Solutions Architect and Sr. Product Extension Manager at Pronovix, and Justin Winter, Business Intelligence Engineer at amazee.io, shared insights into leveraging the potential of LLMs (large language models) to process complex technical documentation.

This project was Inspired by Michael Schmid’s (amazee.io, CTO) demo at LagoonCon 2023. He introduced a private LLM solution and demonstrated a simple implementation, which sparked the idea of proving the solution using very complex, highly technical content sourced from the real world. 

The idea was a private LLM chatbot fed with highly complex, proprietary technical developer documentation for a Pronovix client’s developer portal. With amazee.io’s commitment to privacy and data sovereignty via its advanced CDN and managed WAF, we fit the project’s requirements.

The scope of the proof of concept was limited to one client library package, a DPF C++ Client Library. They were working with raw HTML produced with Doxygen and migrated into Drupal. It was ugly, but it worked! Getting the correct data was crucial to getting good results. As with everything in computing, garbage in, garbage out. 

Justin's focus was to discuss some of the implementation hurdles. He talked about the multi-step process of splitting Markdown into chunks and then processing each chunk with a custom prompt to extract questions/answers, themes, and keywords that then get converted into a vector database. These chunks of text can then be searched when a user asks a question by finding stored text with a similar semantic meaning, as defined by the embeddings of both the question and text chunks. This is known as Retrieval Augmented Generation (RAG) and can dramatically improve the results of LLM output.

Justin then demonstrated how this data is used to build a chat interface. He asked a fundamental question that directly questioned the documentation and received a response. He then requested a more detailed question that required the AI to understand the data. Finally, the chat generated a code example for implementing the library mentioned above.

Justin explained that if you can feed the LLM the context it needs to answer questions, that’s where the real magic is. He also recommended that we consider Chat GPT not as a chatbot but as a search vector. The answers it gives are only as good as the data it has.  This project was done using Llama 3 and Grok LLM. 

If you’re interested in AI hosting, stay tuned for future offerings from Lagoon and amazee.io!

Next up was Bryan Gruneberg, CEO, CTO, and Co-founder at Workshop Orange and Consulting Solution Architect at amazee.io.

He gave a compelling and passionate talk about the Open Web Alliance (OWA), its goals, and how an open source hosting platform like Lagoon aligns perfectly with the OWA’s objectives. Bryan focused on the fact that open source projects deserve to have open source hosting (and that Lagoon is a great choice).

What is the Open Website Alliance?

The Open Website Alliance was founded at the beginning of 2024 to facilitate collaboration between open source content management system projects (specifically Drupal, WordPress, TYPO3, and Joomla) and to drive education and advocacy around open source benefits and principles. Essentially, it’s a community of communities. 

The open web is more than a technology - it is a cause.

The open web is built on freedom. Anyone can learn, build, and advance. It is defined by decentralization - no single person or entity controls the open web. It thrives on inclusion. Anyone can have a home on the open web, regardless of background, identity, ability, wealth, or status. The open web requires participation - it is a shared resource and a shared responsibility, sustained by deliberate choice and collaborative effort. It exists for empowerment. The open web is fueled by the desire for information, connection, and progress and strengthened by individual rights to choice, privacy, and security. 

Bryan walked through the points of the Open Web Manifesto and how Lagoon stacks up. 

It must be designed to protect—not exploit—personal data and public discourse. Lagoon is fully open source, so you can run your platform or hire a company to run your platform, controlling data and discourse as you see fit instead of having to pick a hosting company whose values might not align with yours. 

It must enable the next generation of innovators and entrepreneurs to compete: Scalability and Flexibility are essential. Many hosting providers only allow a couple of types of applications on their platform. If you can put it in a container, Lagoon can deploy it and scale it for any level of traffic.

It must be resilient to a changing world and not controlled by a select few: Only roughly four major Drupal are hosting Platform-as-a-Service (PaaS) offerings. Regarding WordPress, almost 1% of all websites are hosted on Automattic. That’s not a lot of variety. As a fully open source platform, organizations are empowered and even encouraged to run their PaaS with Lagoon.

Bryan wrapped up with an excellent question to think about:

To wrap up the evening, the Lagoon team joined from far and wide to introduce themselves and their work. They also gave a few demos of exciting features being developed for Lagoon.

Matt Swann: UX and UI Updates

Introducing Lagoon User Interface v2! Matt gave us a sneak peek into the upcoming update to the Lagoon UI. Focused on simplicity and functionality, the UI has many new user-friendly features. Users will find quick access to everyday tasks, and overviews will give more information at a glance. 

The UI is managed via Storybook, and a Figma design library is available for the community, which the community is excited about. The team has fostered a culture of testing and evaluating the UI, creating a continuous improvement roadmap. If you’re interested in being a UI tester, let us know! Reach out via email at ux@amazee.io.

UIv2: Problems Tab:

UIv2: Deployments Tab:

UIv2: Environment Overview

Davit Darsavelidze: Lagoon UI Development Improvements

Davit has been working behind the scenes to make the UI development easier. These were his highlights:

UI Component Library

A complete set of React components, combined with NextJS, to make atomic development of future UI functionality more straightforward.

Cypress Testing for the UI

Introducing a full Cypress test suite into the UI and configuring both GitHub actions and local development environments to ensure all development retains functionality, improving iteration speed.

Full Storybook support for the UI, connected to Figma

Similar to Cypress, a development enhancement is needed to allow faster design to code process and improved testing/QA/iteration cycles.

Blaize Kaye: Lagoon Ecosystem/Insights Updates

Blaize has been focusing on the Lagoon Insights suite, which allows users to see various information about their projects in the Lagoon UI. Here’s what he’s been working on:

• Lagoon's Insights feature categorizes valuable data into "Facts" and "Problems," aiding portfolio management.
• Automatically gathered during scans, Insights are processed within Lagoon, facilitating efficient issue identification.
• Custom Insights enrich the system by reporting portfolio-specific information, simplified by recent enhancements.
• New mechanisms, like in-cluster service endpoints, streamline interaction with the Insights API.
• Posting data to Insights APIs is straightforward, allowing seamless integration with various tools.

Blaize has also developed a Lagoon-Remote service for Insights, making it simple to add an Insight. 

Alanna Burke: Lagoon Community Updates

Alanna gave a quick update on the Lagoon community. A major highlight has been the Lagoon Discord, which is growing and thriving - make sure to join if you haven’t yet!

The Lagoon team is still trying to determine what works best for the community regarding Lagoon community hours, so please complete the community survey and help us help you! 

Alanna also highlighted some of the documentation changes in the past year. Some reorganization has been completed to make finding the content you need more accessible. A section on how to Lagoonize your site has been added, a list of client libraries was added, and all releases are now listed in the Lagoon documentation. 

The Lagoon community updates wrapped up with a plea for contribution - it would be wonderful to have everyone contribute, whether directly to the Lagoon project, a supporting project, documentation, or to the most significant need: examples! Lagoon can support almost anything, but help and support are needed by setting up and Lagoonizing sites in various applications, frameworks, and languages to add these to the list of examples. If you’re interested in contributing, check out the Lagoon contribution documentation and say hello in Discord!

Ben Jackson: Build Deploy Tool Iteration, DBaaS Controller Redesign, Lagoon Machinery

Ben is involved in so many aspects of Lagoon that it was hard for him to pick just a few things to discuss. He gave some updates on the behind-the-scenes work to make Lagoon run as smoothly as it does.

• Initially built in BASH, the build deploy tool is now being rewritten in a component-styled architecture that allows for the implementation of individual bits in stages.
• Build logs are now more easily parseable, and smarts have been added for warnings, errors, etc.
• The DBaaS controller was a much-needed overhaul built with the amazee.io platform team. It provides Lagoon environments with their databases, but it has several limitations—it’s non-updatable once deployed, database targets are challenging to auto-select, and it can’t be adapted for other scenarios.
• A future development to watch out for is more composable builds. Quicker and more efficient builds will only create selected services, hot-reloading where applicable, and allow variable-only updates.

Finally, Ben talked about how he and Chris Goodwin (see below) have been working on refactoring Lagoon Services and the Lagoon CLI in Go, using best practices and ensuring that modules are reusable. They’ve taken this rewrite as an opportunity to ensure everything is done correctly, resulting in clean, concise code. 

Brandon Williams: Lagoon and Community

Brandon talked about “eating our dog food.” → The performance tuning for Lagoon core on the amazee.io infrastructure resulted in updates to the MariaDB images so everyone could benefit. 

He has also been providing community support in our Discord channel. He spoke about how Lagoon is seeing community members benefit from our years of support, Lagoon knowledge, and helping one another—which is precisely what Lagoon wants to see in the community support channels!

Brandon has also been working with DDEV to get native support for Lagoon. DDEV is now an excellent option for anyone using Lagoon. If you're interested, read more in the DDEV docs.

Chris Goodwin: Internal-services-test, Lagoon CLI & UI

The wide array of services that Lagoon supports needed to be tested, but what was the best way to do it? A gigantic mega-project with every possible option? No one wants to maintain that. So, the Lagoon team brainstormed and found an efficient way to do this in Go. Enter internal-services-test, an application that tests itself. It uses Go modules for the individual services and, depending on the router's path, triggers an action on a specified endpoint, storing and retrieving data. This application is then used in Lagoon-core testing, Lagoon image testing, and sample projects, and it couldn’t be simpler.

Chris has also been working on the Lagoon CLI, refactoring the codebase, adding support for Organizations, and aligning the CLI better with the API. He’s also worked on the Lagoon UI, adding CRUD functionality for project and environment variables and allowing new environments to be created in the UI. 

Adding an Environment in the Lagoon UI

Managing Organizations in the Lagoon CLI

Managing Environment and Project Variables in the Lagoon UI

The live team updates were a great way to give everyone a close-up and personal of what was in the works and a fun way to wrap up the talks at LagoonCon. 

Wrapping Up LagoonCon 2024

After the LagoonCon talks and presentations, there was time for networking, and many attendees were really excited and already brainstorming new projects with Lagoon.

Jeffrey McGuire (known as “jam” in the Drupal Community) summed up the night perfectly when he commented that since his beginning in the Drupal community in 2005, he has never been to a company-sponsored and run event where the company in question talked so openly and freely about what they were doing, where they were proceeding well, and where they were hitting roadblocks. He commented further that the theory around the open web and open source, which Bryan Gruneberg talked about in his presentation, was achievable in real life, nicely shown and proven through the open source Lagoon project.

Thank you so much to everyone who attended. We hope you had a great time, and we hope to see you both in the Lagoon Discord channel and at LagoonCon Barcelona on September 23, 2024!