[{"data":1,"prerenderedAt":709},["ShallowReactive",2],{"/en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change/":3,"navigation-en-us":36,"banner-en-us":455,"footer-en-us":471,"Igor Wiedler":681,"next-steps-en-us":694},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":8,"content":16,"config":26,"_id":29,"_type":30,"title":31,"_source":32,"_file":33,"_stem":34,"_extension":35},"/en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change","blog",false,"",{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},"This SRE's HAProxy Config Change: An Unexpected Journey","This post is about a wild discovery made while investigating strange behavior from HAProxy. We dive into the pathology, describe how we found it, and share some investigative techniques used along the way.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1749681844/Blog/Hero%20Images/infra-proxy-protocol-wireshark-header.png","https://about.gitlab.com/blog/this-sre-attempted-to-roll-out-an-haproxy-change","https://about.gitlab.com","article","\n                        {\n        \"@context\": \"https://schema.org\",\n        \"@type\": \"Article\",\n        \"headline\": \"This SRE attempted to roll out an HAProxy config change. You won't believe what happened next... \",\n        \"author\": [{\"@type\":\"Person\",\"name\":\"Igor Wiedler\"}],\n        \"datePublished\": \"2021-01-14\",\n      }",{"title":17,"description":10,"authors":18,"heroImage":11,"date":20,"body":21,"category":22,"tags":23},"This SRE attempted to roll out an HAProxy config change. You won't believe what happened next...",[19],"Igor Wiedler","2021-01-14","This blog post was originally published on the GitLab Unfiltered\nblog. It was reviewed and republished on\n2021-02-12.\n\n{: .note .alert-info .text-center}\n\n\n## TL;DR\n\n\n- HAProxy has a `server-state-file` directive that persists some of its\nstate across restarts.\n\n- This state file contains the port of each backend server.\n\n- If an `haproxy.cfg` change modifies the port, the new port will be\noverwritten with the previous one from the state file.\n\n- A workaround is to change the backend server name, so that it is\nconsidered to be a separate server that does not match what is in the state\nfile.\n\n- This has implications for the rollout procedure we use on HAProxy.\n\n\n## Background\n\n\nAll of this occurred in the context of [the gitlab-pages PROXYv2\n\nproject](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11902).\n\n\nThe rollout to staging involves changing the request flow from TCP\nproxying...\n\n```\n                   443                   443                        1443\n[ client ] -> [ google lb ] -> [ fe-pages-01-lb-gstg ] -> [\nweb-pages-01-sv-gstg ]\n      tcp,tls,http         tcp                        tcp            tcp,tls,http\n```\n\n\n... to using the [PROXY\nprotocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt):\n\n```\n                   443                   443                        2443\n[ client ] -> [ google lb ] -> [ fe-pages-01-lb-gstg ] -> [\nweb-pages-01-sv-gstg ]\n      tcp,tls,http         tcp                     proxyv2,tcp       proxyv2,tcp,tls,http\n```\n\n\nThis is done through this change to `/etc/haproxy/haproxy.cfg` on\n\n`fe-pages-01-lb-gstg` (note the port change):\n\n```diff\n\n-    server web-pages-01-sv-gstg\nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080\n\n-    server web-pages-02-sv-gstg\nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080\n\n+    server web-pages-01-sv-gstg\nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n\n+    server web-pages-02-sv-gstg\nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n\n```\n\n\nSeems straightforward enough, let's go ahead and apply that change.\n\n\n## The brokenness\n\n\nAfter applying this change on one of the two `fe-pages` nodes, the requests\nto\n\nthat node start failing.\n\n\nBy retrying a few times via `curl` on the command line, we see this error:\n\n```\n\n➜  ~ curl -vvv https://jarv.staging.gitlab.io/pages-test/\n\n*   Trying 35.229.69.78...\n\n* TCP_NODELAY set\n\n* Connected to jarv.staging.gitlab.io (35.229.69.78) port 443 (#0)\n\n* ALPN, offering h2\n\n* ALPN, offering http/1.1\n\n* successfully set certificate verify locations:\n\n*   CAfile: /etc/ssl/cert.pem\n  CApath: none\n* TLSv1.2 (OUT), TLS handshake, Client hello (1):\n\n* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to\njarv.staging.gitlab.io:443\n\n* Closing connection 0\n\ncurl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to\njarv.staging.gitlab.io:443\n\n```\n\n\nThis looks like some issue in the TLS stack, or possibly with the underlying\n\nconnection. It turns out that `LibreSSL` does not give us much insight into\nthe\n\nunderlying issue here.\n\n\nSo to get a better idea, let's capture a traffic dump on the HAProxy node:\n\n```\n\nsudo tcpdump -v -w \"$(pwd)/$(hostname).$(date +%Y%m%d_%H%M%S).pcap\"\n\n```\n\n\nWhile `tcpdump` is running, we can generate some traffic, then ctrl+c and\npull\n\nthe dump down for further analysis. That `pcap` file can be opened in\nWireshark, and this allows the data to be\n\nexplored and filtered interactively. Here, the first really surprising thing\nhappens:\n\n\n**We do not see any traffic on port 2443.**\n\n\nAt the same time, we _do_ see some traffic on port 1443. But we came here to\nlook at what underlies the LibreSSL error, and what we find\n\nis the following (by filtering for `ip.addr == \u003Cmy external ip>`). We have a\nTCP SYN/ACK, establishing the connection. Followed by the client\n\nsending a TLS \"hello\". After which the server closes the connection with a\nFIN.\n\n\nIn other words, the server is closing the connection on the client.\n\n\n## The early hypotheses\n\n\nSo here come the usual suspects:\n\n\n* Did we modify the correct place in the config file?\n\n* Did we catch all places we need to update in the config?\n\n* Did the HAProxy process parse th econfig successfully?\n\n* Did HAProxy actually reload?\n\n* Is there a difference between reload and restart?\n\n* Did we modify the correct config file?\n\n* Are there old lingering HAProxy processes on the box?\n\n* Are we actually sending traffic to this node?\n\n* Are backend health checks failing?\n\n* Is there anything in the HAProxy logs?\n\n\nNone of these gave any insights whatsoever.\n\n\nIn an effort to reproduce the issue, I ran HAProxy on my local machine with\na\n\nsimilar config, proxying traffic to `web-pages-01-sv-gstg`. To my surprise,\nthis\n\nworked correctly. I tested with different HAProxy versions. It worked\nlocally, but not on\n\n`fe-pages-01`.\n\n\nAt this point I'm stumped. The local config is not identical to gstg, but\nquite\n\nsimilar. What could possibly be the difference?\n\n\n## Digging deeper\n\n\nThis is when I reached out to [Matt Smiley](/company/team#/msmiley) to help\nwith the investigation.\n\n\nWe started off by repeating the experiment. We saw the same results:\n\n\n* Server closes connection after client sends TLS hello\n\n* No traffic from fe-pages to web-pages on port 2443\n\n* Traffic from fe-pages to web-pages on port 1443\n\n\nThe first lead was to look at the packets going to port 1443. What do they\n\ncontain? We see this:\n\n\n![Traffic capture in wireshark showing a TCP FIN and the string QUIT in the\nstream](https://about.gitlab.com/images/blogimages/infra-proxy-protocol-wireshark.png){:\n.shadow.center}\n\nTraffic capture in Wireshark showing a TCP FIN and the string QUIT in the\nstream\n\n{: .note.text-center}\n\n\nThere is mention of `jarv.staging.gitlab.io` which does match what the\nclient sent. And before that there is some really weird preamble:\n\n\n```\n\n\"\\r\\n\\r\\n\\0\\r\\nQUIT\\n\"\n\n```\n\n\nWhat on earth is this? Is it from the PROXY protocol? Let's search [the\n\nspec](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) for the\nword\n\n\"QUIT.\" Nothing.\n\n\nIs this something in the HAProxy source? Searching for \"QUIT\" in the code\n\nreveals some hits, but none that explain this.\n\n\nSo this is a mystery. We leave it for now, and probe in a different\ndirection.\n\n\n## Honing in\n\n\nHow come we are sending traffic to port 1443, when that port is not\nmentioned in\n\n`haproxy.cfg`? Where on earth is HAProxy getting that information from?\n\n\nI suggested running `strace` on HAProxy startup, so that we can see which\nfiles\n\nare being `open`ed. This is a bit tricky to do though, because the process\nis\n\nsystemd-managed.\n\n\nIt turns out that thanks to BPF and [BCC](https://github.com/iovisor/bcc),\nwe\n\ncan actually listen on open events system-wide using the wonderful\n\n[opensnoop](https://github.com/iovisor/bcc/blob/master/tools/opensnoop.py).\nSo we run `opensnoop` and restart `haproxy`, and this is what we see,\nhighlighting the relevant bit:\n\n```\n\niwiedler@fe-pages-01-lb-gstg.c.gitlab-staging-1.internal:~$ sudo\n/usr/share/bcc/tools/opensnoop  -T --name haproxy\n\n\n...\n\n\n24.117171000  16702  haproxy             3   0 /etc/haproxy/haproxy.cfg\n\n...\n\n24.118099000  16702  haproxy             4   0 /etc/haproxy/errors/400.http\n\n...\n\n24.118333000  16702  haproxy             4   0\n/etc/haproxy/cloudflare_ips_v4.lst\n\n...\n\n24.119109000  16702  haproxy             3   0 /etc/haproxy/state/global\n\n```\n\n\nWhat do we have here? `/etc/haproxy/state/global`, this seems oddly\nsuspicious.\n\nWhat could it possibly be? Let's see what this file contains.\n\n```\n\niwiedler@fe-pages-01-lb-gstg.c.gitlab-staging-1.internal:~$ sudo cat\n/etc/haproxy/state/global\n\n\n1\n\n# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state\nsrv_uweight srv_iweight srv_time_since_last_change srv_check_status\nsrv_check_result srv_check_health srv_check_state srv_agent_state\nbk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord\n\n5 pages_http 1 web-pages-01-sv-gstg 10.224.26.2 2 0 1 1 21134 15 3 4 6 0 0 0\nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal 1080 -\n\n5 pages_http 2 web-pages-02-sv-gstg 10.224.26.3 2 0 1 1 20994 15 3 4 6 0 0 0\nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal 1080 -\n\n6 pages_https 1 web-pages-01-sv-gstg 10.224.26.2 2 0 1 1 21134 15 3 4 6 0 0\n0 web-pages-01-sv-gstg.c.gitlab-staging-1.internal 1443 -\n\n6 pages_https 2 web-pages-02-sv-gstg 10.224.26.3 2 0 1 1 20994 15 3 4 6 0 0\n0 web-pages-02-sv-gstg.c.gitlab-staging-1.internal 1443 -\n\n```\n\n\nIt appears we are storing some metadata for each backend server, including\nits old port number!\n\n\nNow, looking again in `haproxy.cfg`, we see:\n\n```\n\nglobal\n    ...\n    server-state-file /etc/haproxy/state/global\n```\n\n\nSo we are using the\n\n[`server-state-file`](https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#server-state-file)\n\ndirective. This will persist server state across HAProxy restarts. That is\n\nuseful to keep metadata consistent, such as whether a server was marked as\n\nMAINT.\n\n\n**However, it appears to be clobbering the port from `haproxy.cfg`!**\n\n\nThe suspected behavior is:\n\n\n* HAProxy is running with the old config: `web-pages-01-sv-gstg`, `1443`\n\n* `haproxy.cfg` is updated with the new config: `web-pages-01-sv-gstg`,\n`2443`, `send-proxy-v2`\n\n* HAProxy reload is initiated\n\n* HAProxy writes out the state to `/etc/haproxy/state/global` (including the\nold port of each backend server)\n\n* HAProxy starts up, reads `haproxy.cfg`, initializes itself with the new\nconfig: `web-pages-01-sv-gstg`, `2443`, `send-proxy-v2`\n\n* HAProxy reads the state from `/etc/haproxy/state/global`, matches on the\nbackend server `web-pages-01-sv-gstg`, and overrides all values, including\nthe port!\n\n\nThe result is that we are now attempting to send PROXYv2 traffic to the TLS\nport.\n\n\n## The workaround\n\n\nTo validate the theory and develop a potential workaround, we modify\n\n`haproxy.cfg` to use a different backend server name.\n\n\nThe new diff is:\n\n```diff\n\n-    server web-pages-01-sv-gstg        \nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080\n\n-    server web-pages-02-sv-gstg        \nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080\n\n+    server web-pages-01-sv-gstg-proxyv2\nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n\n+    server web-pages-02-sv-gstg-proxyv2\nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n\n```\n\n\nWith this config change in place, we reload HAProxy and indeed, it is now\n\nserving traffic correctly. See [the merge request fixing\nit](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/merge_requests/261).\n\n\n## A follow-up on those `QUIT` bytes\n\n\nNow, what is up with that `QUIT` message? Is it part of the PROXY protocol?\nRemember, searching [the\n\nspec](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) for that\n\nstring did not find any matches. However, Matt actually read the spec, and\nfound this section on version 2 of\n\nthe protocol:\n\n```\n\nThe binary header format starts with a constant 12 bytes block containing\nthe\n\nprotocol signature :\n\n   \\x0D \\x0A \\x0D \\x0A \\x00 \\x0D \\x0A \\x51 \\x55 \\x49 \\x54 \\x0A\n```\n\n\nThose are indeed the bytes that make up \"\\r\\n\\r\\n\\0\\r\\nQUIT\\n\". Slightly\nless mnemonic than the header from text-based version 1 of the protocol:\n\n```\n\n- a string identifying the protocol : \"PROXY\" ( \\x50 \\x52 \\x4F \\x58 \\x59 )\n  Seeing this string indicates that this is version 1 of the protocol.\n```\n\n\nWell, I suppose that explains it.\n\n\nI believe our work here is done. Don't forget to like and subscribe!\n","engineering",[24,25],"production","inside GitLab",{"slug":27,"featured":6,"template":28},"this-sre-attempted-to-roll-out-an-haproxy-change","BlogPost","content:en-us:blog:this-sre-attempted-to-roll-out-an-haproxy-change.yml","yaml","This Sre Attempted To Roll Out An Haproxy Change","content","en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change.yml","en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change","yml",{"_path":37,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"data":39,"_id":451,"_type":30,"title":452,"_source":32,"_file":453,"_stem":454,"_extension":35},"/shared/en-us/main-navigation","en-us",{"logo":40,"freeTrial":45,"sales":50,"login":55,"items":60,"search":392,"minimal":423,"duo":442},{"config":41},{"href":42,"dataGaName":43,"dataGaLocation":44},"/","gitlab logo","header",{"text":46,"config":47},"Get free trial",{"href":48,"dataGaName":49,"dataGaLocation":44},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":51,"config":52},"Talk to sales",{"href":53,"dataGaName":54,"dataGaLocation":44},"/sales/","sales",{"text":56,"config":57},"Sign in",{"href":58,"dataGaName":59,"dataGaLocation":44},"https://gitlab.com/users/sign_in/","sign in",[61,105,203,208,313,373],{"text":62,"config":63,"cards":65,"footer":88},"Platform",{"dataNavLevelOne":64},"platform",[66,72,80],{"title":62,"description":67,"link":68},"The most comprehensive AI-powered DevSecOps Platform",{"text":69,"config":70},"Explore our Platform",{"href":71,"dataGaName":64,"dataGaLocation":44},"/platform/",{"title":73,"description":74,"link":75},"GitLab Duo (AI)","Build software faster with AI at every stage of development",{"text":76,"config":77},"Meet GitLab Duo",{"href":78,"dataGaName":79,"dataGaLocation":44},"/gitlab-duo/","gitlab duo ai",{"title":81,"description":82,"link":83},"Why GitLab","10 reasons why Enterprises choose GitLab",{"text":84,"config":85},"Learn more",{"href":86,"dataGaName":87,"dataGaLocation":44},"/why-gitlab/","why gitlab",{"title":89,"items":90},"Get started with",[91,96,101],{"text":92,"config":93},"Platform Engineering",{"href":94,"dataGaName":95,"dataGaLocation":44},"/solutions/platform-engineering/","platform engineering",{"text":97,"config":98},"Developer Experience",{"href":99,"dataGaName":100,"dataGaLocation":44},"/developer-experience/","Developer experience",{"text":102,"config":103},"MLOps",{"href":104,"dataGaName":102,"dataGaLocation":44},"/topics/devops/the-role-of-ai-in-devops/",{"text":106,"left":107,"config":108,"link":110,"lists":114,"footer":185},"Product",true,{"dataNavLevelOne":109},"solutions",{"text":111,"config":112},"View all Solutions",{"href":113,"dataGaName":109,"dataGaLocation":44},"/solutions/",[115,140,164],{"title":116,"description":117,"link":118,"items":123},"Automation","CI/CD and automation to accelerate deployment",{"config":119},{"icon":120,"href":121,"dataGaName":122,"dataGaLocation":44},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[124,128,132,136],{"text":125,"config":126},"CI/CD",{"href":127,"dataGaLocation":44,"dataGaName":125},"/solutions/continuous-integration/",{"text":129,"config":130},"AI-Assisted Development",{"href":78,"dataGaLocation":44,"dataGaName":131},"AI assisted development",{"text":133,"config":134},"Source Code Management",{"href":135,"dataGaLocation":44,"dataGaName":133},"/solutions/source-code-management/",{"text":137,"config":138},"Automated Software Delivery",{"href":121,"dataGaLocation":44,"dataGaName":139},"Automated software delivery",{"title":141,"description":142,"link":143,"items":148},"Security","Deliver code faster without compromising security",{"config":144},{"href":145,"dataGaName":146,"dataGaLocation":44,"icon":147},"/solutions/security-compliance/","security and compliance","ShieldCheckLight",[149,154,159],{"text":150,"config":151},"Application Security Testing",{"href":152,"dataGaName":153,"dataGaLocation":44},"/solutions/application-security-testing/","Application security testing",{"text":155,"config":156},"Software Supply Chain Security",{"href":157,"dataGaLocation":44,"dataGaName":158},"/solutions/supply-chain/","Software supply chain security",{"text":160,"config":161},"Software Compliance",{"href":162,"dataGaName":163,"dataGaLocation":44},"/solutions/software-compliance/","software compliance",{"title":165,"link":166,"items":171},"Measurement",{"config":167},{"icon":168,"href":169,"dataGaName":170,"dataGaLocation":44},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[172,176,180],{"text":173,"config":174},"Visibility & Measurement",{"href":169,"dataGaLocation":44,"dataGaName":175},"Visibility and Measurement",{"text":177,"config":178},"Value Stream Management",{"href":179,"dataGaLocation":44,"dataGaName":177},"/solutions/value-stream-management/",{"text":181,"config":182},"Analytics & Insights",{"href":183,"dataGaLocation":44,"dataGaName":184},"/solutions/analytics-and-insights/","Analytics and insights",{"title":186,"items":187},"GitLab for",[188,193,198],{"text":189,"config":190},"Enterprise",{"href":191,"dataGaLocation":44,"dataGaName":192},"/enterprise/","enterprise",{"text":194,"config":195},"Small Business",{"href":196,"dataGaLocation":44,"dataGaName":197},"/small-business/","small business",{"text":199,"config":200},"Public Sector",{"href":201,"dataGaLocation":44,"dataGaName":202},"/solutions/public-sector/","public sector",{"text":204,"config":205},"Pricing",{"href":206,"dataGaName":207,"dataGaLocation":44,"dataNavLevelOne":207},"/pricing/","pricing",{"text":209,"config":210,"link":212,"lists":216,"feature":300},"Resources",{"dataNavLevelOne":211},"resources",{"text":213,"config":214},"View all resources",{"href":215,"dataGaName":211,"dataGaLocation":44},"/resources/",[217,250,272],{"title":218,"items":219},"Getting started",[220,225,230,235,240,245],{"text":221,"config":222},"Install",{"href":223,"dataGaName":224,"dataGaLocation":44},"/install/","install",{"text":226,"config":227},"Quick start guides",{"href":228,"dataGaName":229,"dataGaLocation":44},"/get-started/","quick setup checklists",{"text":231,"config":232},"Learn",{"href":233,"dataGaLocation":44,"dataGaName":234},"https://university.gitlab.com/","learn",{"text":236,"config":237},"Product documentation",{"href":238,"dataGaName":239,"dataGaLocation":44},"https://docs.gitlab.com/","product documentation",{"text":241,"config":242},"Best practice videos",{"href":243,"dataGaName":244,"dataGaLocation":44},"/getting-started-videos/","best practice videos",{"text":246,"config":247},"Integrations",{"href":248,"dataGaName":249,"dataGaLocation":44},"/integrations/","integrations",{"title":251,"items":252},"Discover",[253,258,262,267],{"text":254,"config":255},"Customer success stories",{"href":256,"dataGaName":257,"dataGaLocation":44},"/customers/","customer success stories",{"text":259,"config":260},"Blog",{"href":261,"dataGaName":5,"dataGaLocation":44},"/blog/",{"text":263,"config":264},"Remote",{"href":265,"dataGaName":266,"dataGaLocation":44},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"text":268,"config":269},"TeamOps",{"href":270,"dataGaName":271,"dataGaLocation":44},"/teamops/","teamops",{"title":273,"items":274},"Connect",[275,280,285,290,295],{"text":276,"config":277},"GitLab Services",{"href":278,"dataGaName":279,"dataGaLocation":44},"/services/","services",{"text":281,"config":282},"Community",{"href":283,"dataGaName":284,"dataGaLocation":44},"/community/","community",{"text":286,"config":287},"Forum",{"href":288,"dataGaName":289,"dataGaLocation":44},"https://forum.gitlab.com/","forum",{"text":291,"config":292},"Events",{"href":293,"dataGaName":294,"dataGaLocation":44},"/events/","events",{"text":296,"config":297},"Partners",{"href":298,"dataGaName":299,"dataGaLocation":44},"/partners/","partners",{"backgroundColor":301,"textColor":302,"text":303,"image":304,"link":308},"#2f2a6b","#fff","Insights for the future of software development",{"altText":305,"config":306},"the source promo card",{"src":307},"/images/navigation/the-source-promo-card.svg",{"text":309,"config":310},"Read the latest",{"href":311,"dataGaName":312,"dataGaLocation":44},"/the-source/","the source",{"text":314,"config":315,"lists":317},"Company",{"dataNavLevelOne":316},"company",[318],{"items":319},[320,325,331,333,338,343,348,353,358,363,368],{"text":321,"config":322},"About",{"href":323,"dataGaName":324,"dataGaLocation":44},"/company/","about",{"text":326,"config":327,"footerGa":330},"Jobs",{"href":328,"dataGaName":329,"dataGaLocation":44},"/jobs/","jobs",{"dataGaName":329},{"text":291,"config":332},{"href":293,"dataGaName":294,"dataGaLocation":44},{"text":334,"config":335},"Leadership",{"href":336,"dataGaName":337,"dataGaLocation":44},"/company/team/e-group/","leadership",{"text":339,"config":340},"Team",{"href":341,"dataGaName":342,"dataGaLocation":44},"/company/team/","team",{"text":344,"config":345},"Handbook",{"href":346,"dataGaName":347,"dataGaLocation":44},"https://handbook.gitlab.com/","handbook",{"text":349,"config":350},"Investor relations",{"href":351,"dataGaName":352,"dataGaLocation":44},"https://ir.gitlab.com/","investor relations",{"text":354,"config":355},"Trust Center",{"href":356,"dataGaName":357,"dataGaLocation":44},"/security/","trust center",{"text":359,"config":360},"AI Transparency Center",{"href":361,"dataGaName":362,"dataGaLocation":44},"/ai-transparency-center/","ai transparency center",{"text":364,"config":365},"Newsletter",{"href":366,"dataGaName":367,"dataGaLocation":44},"/company/contact/","newsletter",{"text":369,"config":370},"Press",{"href":371,"dataGaName":372,"dataGaLocation":44},"/press/","press",{"text":374,"config":375,"lists":376},"Contact us",{"dataNavLevelOne":316},[377],{"items":378},[379,382,387],{"text":51,"config":380},{"href":53,"dataGaName":381,"dataGaLocation":44},"talk to sales",{"text":383,"config":384},"Get help",{"href":385,"dataGaName":386,"dataGaLocation":44},"/support/","get help",{"text":388,"config":389},"Customer portal",{"href":390,"dataGaName":391,"dataGaLocation":44},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":393,"login":394,"suggestions":401},"Close",{"text":395,"link":396},"To search repositories and projects, login to",{"text":397,"config":398},"gitlab.com",{"href":58,"dataGaName":399,"dataGaLocation":400},"search login","search",{"text":402,"default":403},"Suggestions",[404,406,410,412,416,420],{"text":73,"config":405},{"href":78,"dataGaName":73,"dataGaLocation":400},{"text":407,"config":408},"Code Suggestions (AI)",{"href":409,"dataGaName":407,"dataGaLocation":400},"/solutions/code-suggestions/",{"text":125,"config":411},{"href":127,"dataGaName":125,"dataGaLocation":400},{"text":413,"config":414},"GitLab on AWS",{"href":415,"dataGaName":413,"dataGaLocation":400},"/partners/technology-partners/aws/",{"text":417,"config":418},"GitLab on Google Cloud",{"href":419,"dataGaName":417,"dataGaLocation":400},"/partners/technology-partners/google-cloud-platform/",{"text":421,"config":422},"Why GitLab?",{"href":86,"dataGaName":421,"dataGaLocation":400},{"freeTrial":424,"mobileIcon":429,"desktopIcon":434,"secondaryButton":437},{"text":425,"config":426},"Start free trial",{"href":427,"dataGaName":49,"dataGaLocation":428},"https://gitlab.com/-/trials/new/","nav",{"altText":430,"config":431},"Gitlab Icon",{"src":432,"dataGaName":433,"dataGaLocation":428},"/images/brand/gitlab-logo-tanuki.svg","gitlab icon",{"altText":430,"config":435},{"src":436,"dataGaName":433,"dataGaLocation":428},"/images/brand/gitlab-logo-type.svg",{"text":438,"config":439},"Get Started",{"href":440,"dataGaName":441,"dataGaLocation":428},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/compare/gitlab-vs-github/","get started",{"freeTrial":443,"mobileIcon":447,"desktopIcon":449},{"text":444,"config":445},"Learn more about GitLab Duo",{"href":78,"dataGaName":446,"dataGaLocation":428},"gitlab duo",{"altText":430,"config":448},{"src":432,"dataGaName":433,"dataGaLocation":428},{"altText":430,"config":450},{"src":436,"dataGaName":433,"dataGaLocation":428},"content:shared:en-us:main-navigation.yml","Main Navigation","shared/en-us/main-navigation.yml","shared/en-us/main-navigation",{"_path":456,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"title":457,"button":458,"image":462,"config":466,"_id":468,"_type":30,"_source":32,"_file":469,"_stem":470,"_extension":35},"/shared/en-us/banner","is now in public beta!",{"text":84,"config":459},{"href":460,"dataGaName":461,"dataGaLocation":44},"/gitlab-duo/agent-platform/","duo banner",{"altText":463,"config":464},"GitLab Duo Agent Platform",{"src":465},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1753720689/somrf9zaunk0xlt7ne4x.svg",{"layout":467},"release","content:shared:en-us:banner.yml","shared/en-us/banner.yml","shared/en-us/banner",{"_path":472,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"data":473,"_id":677,"_type":30,"title":678,"_source":32,"_file":679,"_stem":680,"_extension":35},"/shared/en-us/main-footer",{"text":474,"source":475,"edit":481,"contribute":486,"config":491,"items":496,"minimal":669},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":476,"config":477},"View page source",{"href":478,"dataGaName":479,"dataGaLocation":480},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":482,"config":483},"Edit this page",{"href":484,"dataGaName":485,"dataGaLocation":480},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":487,"config":488},"Please contribute",{"href":489,"dataGaName":490,"dataGaLocation":480},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":492,"facebook":493,"youtube":494,"linkedin":495},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[497,520,576,605,639],{"title":62,"links":498,"subMenu":503},[499],{"text":500,"config":501},"DevSecOps platform",{"href":71,"dataGaName":502,"dataGaLocation":480},"devsecops platform",[504],{"title":204,"links":505},[506,510,515],{"text":507,"config":508},"View plans",{"href":206,"dataGaName":509,"dataGaLocation":480},"view plans",{"text":511,"config":512},"Why Premium?",{"href":513,"dataGaName":514,"dataGaLocation":480},"/pricing/premium/","why premium",{"text":516,"config":517},"Why Ultimate?",{"href":518,"dataGaName":519,"dataGaLocation":480},"/pricing/ultimate/","why ultimate",{"title":521,"links":522},"Solutions",[523,528,530,532,537,542,546,549,553,558,560,563,566,571],{"text":524,"config":525},"Digital transformation",{"href":526,"dataGaName":527,"dataGaLocation":480},"/topics/digital-transformation/","digital transformation",{"text":150,"config":529},{"href":152,"dataGaName":150,"dataGaLocation":480},{"text":139,"config":531},{"href":121,"dataGaName":122,"dataGaLocation":480},{"text":533,"config":534},"Agile development",{"href":535,"dataGaName":536,"dataGaLocation":480},"/solutions/agile-delivery/","agile delivery",{"text":538,"config":539},"Cloud transformation",{"href":540,"dataGaName":541,"dataGaLocation":480},"/topics/cloud-native/","cloud transformation",{"text":543,"config":544},"SCM",{"href":135,"dataGaName":545,"dataGaLocation":480},"source code management",{"text":125,"config":547},{"href":127,"dataGaName":548,"dataGaLocation":480},"continuous integration & delivery",{"text":550,"config":551},"Value stream management",{"href":179,"dataGaName":552,"dataGaLocation":480},"value stream management",{"text":554,"config":555},"GitOps",{"href":556,"dataGaName":557,"dataGaLocation":480},"/solutions/gitops/","gitops",{"text":189,"config":559},{"href":191,"dataGaName":192,"dataGaLocation":480},{"text":561,"config":562},"Small business",{"href":196,"dataGaName":197,"dataGaLocation":480},{"text":564,"config":565},"Public sector",{"href":201,"dataGaName":202,"dataGaLocation":480},{"text":567,"config":568},"Education",{"href":569,"dataGaName":570,"dataGaLocation":480},"/solutions/education/","education",{"text":572,"config":573},"Financial services",{"href":574,"dataGaName":575,"dataGaLocation":480},"/solutions/finance/","financial services",{"title":209,"links":577},[578,580,582,584,587,589,591,593,595,597,599,601,603],{"text":221,"config":579},{"href":223,"dataGaName":224,"dataGaLocation":480},{"text":226,"config":581},{"href":228,"dataGaName":229,"dataGaLocation":480},{"text":231,"config":583},{"href":233,"dataGaName":234,"dataGaLocation":480},{"text":236,"config":585},{"href":238,"dataGaName":586,"dataGaLocation":480},"docs",{"text":259,"config":588},{"href":261,"dataGaName":5,"dataGaLocation":480},{"text":254,"config":590},{"href":256,"dataGaName":257,"dataGaLocation":480},{"text":263,"config":592},{"href":265,"dataGaName":266,"dataGaLocation":480},{"text":276,"config":594},{"href":278,"dataGaName":279,"dataGaLocation":480},{"text":268,"config":596},{"href":270,"dataGaName":271,"dataGaLocation":480},{"text":281,"config":598},{"href":283,"dataGaName":284,"dataGaLocation":480},{"text":286,"config":600},{"href":288,"dataGaName":289,"dataGaLocation":480},{"text":291,"config":602},{"href":293,"dataGaName":294,"dataGaLocation":480},{"text":296,"config":604},{"href":298,"dataGaName":299,"dataGaLocation":480},{"title":314,"links":606},[607,609,611,613,615,617,619,623,628,630,632,634],{"text":321,"config":608},{"href":323,"dataGaName":316,"dataGaLocation":480},{"text":326,"config":610},{"href":328,"dataGaName":329,"dataGaLocation":480},{"text":334,"config":612},{"href":336,"dataGaName":337,"dataGaLocation":480},{"text":339,"config":614},{"href":341,"dataGaName":342,"dataGaLocation":480},{"text":344,"config":616},{"href":346,"dataGaName":347,"dataGaLocation":480},{"text":349,"config":618},{"href":351,"dataGaName":352,"dataGaLocation":480},{"text":620,"config":621},"Sustainability",{"href":622,"dataGaName":620,"dataGaLocation":480},"/sustainability/",{"text":624,"config":625},"Diversity, inclusion and belonging (DIB)",{"href":626,"dataGaName":627,"dataGaLocation":480},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":354,"config":629},{"href":356,"dataGaName":357,"dataGaLocation":480},{"text":364,"config":631},{"href":366,"dataGaName":367,"dataGaLocation":480},{"text":369,"config":633},{"href":371,"dataGaName":372,"dataGaLocation":480},{"text":635,"config":636},"Modern Slavery Transparency Statement",{"href":637,"dataGaName":638,"dataGaLocation":480},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"title":640,"links":641},"Contact Us",[642,645,647,649,654,659,664],{"text":643,"config":644},"Contact an expert",{"href":53,"dataGaName":54,"dataGaLocation":480},{"text":383,"config":646},{"href":385,"dataGaName":386,"dataGaLocation":480},{"text":388,"config":648},{"href":390,"dataGaName":391,"dataGaLocation":480},{"text":650,"config":651},"Status",{"href":652,"dataGaName":653,"dataGaLocation":480},"https://status.gitlab.com/","status",{"text":655,"config":656},"Terms of use",{"href":657,"dataGaName":658,"dataGaLocation":480},"/terms/","terms of use",{"text":660,"config":661},"Privacy statement",{"href":662,"dataGaName":663,"dataGaLocation":480},"/privacy/","privacy statement",{"text":665,"config":666},"Cookie preferences",{"dataGaName":667,"dataGaLocation":480,"id":668,"isOneTrustButton":107},"cookie preferences","ot-sdk-btn",{"items":670},[671,673,675],{"text":655,"config":672},{"href":657,"dataGaName":658,"dataGaLocation":480},{"text":660,"config":674},{"href":662,"dataGaName":663,"dataGaLocation":480},{"text":665,"config":676},{"dataGaName":667,"dataGaLocation":480,"id":668,"isOneTrustButton":107},"content:shared:en-us:main-footer.yml","Main Footer","shared/en-us/main-footer.yml","shared/en-us/main-footer",[682],{"_path":683,"_dir":684,"_draft":6,"_partial":6,"_locale":7,"content":685,"config":689,"_id":691,"_type":30,"title":19,"_source":32,"_file":692,"_stem":693,"_extension":35},"/en-us/blog/authors/igor-wiedler","authors",{"name":19,"config":686},{"headshot":687,"ctfId":688},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749681841/Blog/Author%20Headshots/igorwwwwwwwwwwwwwwwwwwww-headshot.png","igorwwwwwwwwwwwwwwwwwwww",{"template":690},"BlogAuthor","content:en-us:blog:authors:igor-wiedler.yml","en-us/blog/authors/igor-wiedler.yml","en-us/blog/authors/igor-wiedler",{"_path":695,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"header":696,"eyebrow":697,"blurb":698,"button":699,"secondaryButton":703,"_id":705,"_type":30,"title":706,"_source":32,"_file":707,"_stem":708,"_extension":35},"/shared/en-us/next-steps","Start shipping better software faster","50%+ of the Fortune 100 trust GitLab","See what your team can do with the intelligent\n\n\nDevSecOps platform.\n",{"text":46,"config":700},{"href":701,"dataGaName":49,"dataGaLocation":702},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":51,"config":704},{"href":53,"dataGaName":54,"dataGaLocation":702},"content:shared:en-us:next-steps.yml","Next Steps","shared/en-us/next-steps.yml","shared/en-us/next-steps",1755803081799]