Planned solution (the “nice” one, by the book): route all outbound traffic through a corporate proxy stack (NLB > proxy in EC2) and let security whitelist only the proxy IP. On the Greengrass side, AWS has recipes for this - in theory, you set a few proxy variables in the merge config of your Nucleus and you’re done.

Reality #1: the docs are vague. The “just add a couple of lines to the Nucleus config” approach reads fine on paper, but in the field, you need to ensure the usual proxy envs are present everywhere: HTTP_PROXY, HTTPS_PROXY, NO_PROXY (and uppercase variants). Setting them only in the nucleus isn’t enough if components ignore the nucleus-environment.

Reality #2: most public components don’t play by the rules. They bypass the proxy and try to send traffic directly. Worse - there’s a floating Systems Manager bug where its proxy variables get reset after a Greengrass restart. So you can deploy, verify, sleep, and then a reboot turns a tidy, secure device into a leaking sieve. Or simply locks you out of your edge device as Systems Manager is hitting the firewall.

What I did (blunt solution until AWS addresses this issue):

• Ensure proxy envs are explicitly injected into each public component (don’t rely solely on nucleus inheritance).

• Manually create a drop-in config for the proxy variables. Greengrass won’t overwrite it on restart, so it sticks.

• Recommended hack, echoed by an AWS architect: introduce a small “service component” that applies all critical settings (proxy, certs, env vars) and declare it as a dependency for every other component. That way, no other component even starts until the environment is sane.

And this is a perfect example of edge reality: the architecture diagrams look pretty, but the edge prefers surprises.

Filed the relevant bug/support case with AWS; until it’s fixed, treat Systems Manager’s public components like “cattle that occasionally forget they are behind a wall” and hard-enforce proxy config locally.

Note for myself and my brothers and sisters on the edge: always test your components against GG restarts, validate env propagation per component. It might save some night shifts.

When the components, the devices, were ranged in columns before me,

When I was shown the recipes and groups, to add, divide, and measure them,

When I sitting heard the architect where he lectured with much applause in the lecture-room,

How soon unaccountable I became tired and sick,

Till rising and gliding out I wander’d off by myself,

In the mystical moist night-air, and from time to time,

Look’d up in perfect silence at the CloudWatch.

To my dear W.W.

This is my intro article in the “Leaves of Greengrass“ cycle. In it, I would like to share my knowledge and experience with AWS Greengrass and some extra pains someone might find entertaining and maybe even useful.

I’ve been living with AWS Greengrass for a while now. Depending on the day, it feels like either a very cool and serious project or a science experiment someone duct-taped together after too much Red Bull, and now I have to make it live.

On paper, Greengrass looks clean: “here are the components, here are the groups, here’s how it all connects.” In reality, you’re spelunking through logs in two or three different places, spewing deployments like a machine gun, and considering the next SDK update as a major threat.

The project I’m working on is far from what it should be on paper or ones might do by following tutorials. It is a production environment where internet connection is a luxury and costs a fortune, downtime is not an option, and every second of data is measured in terabytes.

Sounds badass, you’d say! Sounds like a minefield, I’d say.

So why write about this?

• Because the docs are sterile, and I want to show the dirt. At least, i wanted someone to show it to me back in the days.

• Because behind many fixes there’s not just a technical answer, but a survival story.

• Because at 2 a.m., staring at Greengrass logs, it helps to know you’re not alone.

What Leaves of Greengrass will cover:

• Components that refuse to start and won’t tell you why.

• Recipes and groups that look simple in diagrams but turn into hell in practice.

• The small victories when, after twenty failed attempts, something finally runs.

Anyway, I write it as my field notes. Part war stories, part technical breakdowns.

If you love polished AWS whitepapers, this will hurt.

If you want to know what Greengrass looks like in real life, welcome.