Yes, there is a long migration period (almost 2 years). Yes, if you are paying for support, they will help you a lot with the migration.
But if you are using a (costly) managed service it's because you don't want the hurdle of managing the service. Migrating off something is instead probably the most time consuming thing in a service lifecycle. So, even if it is normal to sunset products and people deal with it all the time, it's still a big PITA if you are affected.
> But if you are using a (costly) managed service it's because you don't want the hurdle of managing the service.
On the other hand part of the trade offs of using such services is that you have no control of when they will become unavailable. Its not just normal, its expected.
Most SMEs do not even seem to have backups with another provider.
I cannot see EC2 being shut down but other services might be, and there are all sorts of other issues. The idea that a cloud provider will just take care of everything is delusional. You still have work to do. It might be less work, but its there.
There was an Australian pension provider that recently had an issue with their main cloud account and services shut down. Luckily the regulator required they had backups with a different provider so they were able to restore everything. This is a business running something like AUD 125bn of other people's money.
it's not just managed service where you can move to your self hosted version when they stop managing it, it's a fully proprietary stack that's just no longer available to use.
The one service that is not dead yet. But is smelling funny is CodePipeline. It is so bad and they’ve already discontinued CodeCommit (the very bad hosted git service), Cloud9 (hosted IDE that is based on VSCode even though they would never admit it externally) and CodeStar.
AFAIK cloud9 was never vscode based, it was a dutch startup acquisition when hosted RDEs were looking popular https://siliconcanals.com/amazon-acquires-cloud9/. I believe theyre busy working on similar-but-not-obviously related remote environments like AWS Workspaces.
Re: existing Code-asterisk _services_ see CodeCatalyst at a recent (in progress) attempt to move to a more unified user-centric _product_.
And for downthread Cognito, Id be surprised if it ever goes away. i think theres some unification/deconfliction of capabilities that needs to happen with Federate. But Id expect one or both of those to stick around as a proxy pool of user identities for apps. The more interesting, and huge change to how AWS fundamentally works, is Identity Center(IdC). Tying back to cognito and CodeCatalyst IdC should _finally_ bring user identities, and your corp IdP, in to AWS and actually be useful across existing services. This _could_ catch AWS up to goog cloud and OCI and (I assume) azure on identity management.
Edit: ex AWS, but all if the above based on publicly available information.
They can’t really get rid of Cognito. It’s the authentication/authorization platform for external users. There is no migration path that will seamlessly transfer user names and passwords to other mainstream services like CodeCommit.
If you run internal apps on ECS, cognito effectively gives you beyondcorp for free. Being able to put cognito as a rule in your load balancer is choice. I wouldn't use it for anything else, though.
A company with Amazons resources could keep these fringe services around as long as there are paying customers. Just increase the price of the service to match the costs of keeping the legacy service around.
Three problems with this approach.
1. The ongoing “ktlo” (keep the lights on) maintenance cost for a minimal public aws service is on the order of 6 SDEs. Theres a constant stream of security and integration improvements like new IAM policy features, KMS handling, billing & metering, console chrome, docs, etc. These are generally nonnegotiable to keep _any_ sort of consistency across AWS.
This is discounting most operations/operational support, as you cant practically scale on oncall rota that small and your example is probably referring to a small customer base and a very infrastructure light service. You _could_ move that to a cheaper dev center & more shared ops, but that in itself is a lot of work.
2. Theres a lot of overlap between these services. Its often not complete, and theyll have different approaches. But fundamentally Amazons distributed nature means multiple orgs will solve similar problems that are adjacent but outside their focus, and then realize that “product” is someones elses better owned feature. Culling needs tk happen at that user experience level sooner or later. The linked blog post is a good example.
3. “Pay the average cost” is going to be shocking to see how much some of these cost to run. This cost increase is going to feel like extortion for customers who _already use other service_ and probably pay a lot more. Earning ongoing negative feedback over a minor service is not worth it. Nearly all services are priced based on a multi year forward looking marginal cost model. I would very much expect the average cost to be 10x the customer price for a small service like app mesh that never “reached scale” or got their P&L together. Then add in those dev salaries I mentioned.
Most importantly those ktlo dev hours in point #1 are an opportunity cost for AWS. They should be spent on new revenue generating work, not cost reduction, and certajnly not a dead end. More than 10 years ago the rough rule of thumb was that a dev year of work should be targeting $12M/yr in revenue generation. I know we didnt consistently hit that, but thats the kind of benchmark that would be used to evaluate staffing. In AWS today if the overall product doesnt have a line of sight to $500M its not a viable innestment.
You’re forgetting one important point. No one who has any interest in their career inside or outside of Amazon wants to be on a service team for a dead service. That would look horrible on both their promo doc internally and when trying to get through a behavioral interview externally.
It definitely scales of complexity/surface area, and Id say that straight number is a lower bounds fixed cost that you might see from an unsuccessful “small” service. In larger orgs its better a percentage, because its relative to overall investment/growth. My personal goal would be 20% ktlo, usually much closer to 40% in practice. All based on my personal experience doing the cloud thing across 2 companies for 15 years or so.
And I will give credit to AMZN/AWS, of old at least, that the above would be common knowledge with an expectation of much more org specific details for anyone sr manager/PE through VP whod be involved in roadmaps, HC allocation, etc.
are we assuming only 9-5? 2.3 SDE could do that, but i don't get 6. a 24/7 operation needs 2FTE and 1 PTE per shift, and there are three shifts.
And i suppose "6 SDEs" for a service that has... what number of MAU or whatever?
me and a great friend of mine split our roles with our "projects", he deals with paid clients, and i deal with our required services (email, matrix, nextcloud, pastebins, misskey, PBXen, wireguard/VPN/Proxying...) and there's just two of us. I can't even think of the last time we had downtime, except at the tail-end of last year, one or the other of us caused a fault in proxmox, where commands were no longer issuing, and we had to do a reboot and fsck, the drill.
I understand at "AWS" scale obviously there's going to be more fires than 1 or two people can put out, but 6 SDEs is over 2 million in compensation all told for amazon, right?
So i guess what i am asking for is context. I've run stuff with hundreds of millions of monthly unique (for a paycheck). That company had over a billion uniques across all of their verticals 14 years ago. One network admin (like senior, but still). When i was there, there were 4 DBAs. there was 1 who could deal with developers, the oracle/postgresql guy who is one of my favorite humans ever - he could convert C-level grunts into sql queries or whatever it is they do. Then there was a part time mariadb; and then a full time Oracle DBA for just 1 site. I'm really straining to remember who else was basically infra/ops and it was less than 6[^1] of us for the whole company. We did have a NOC in another country but prior to my arrival all they were really trained to know how to do was page the correct person in the US. Before i left, 3 of the NOC people moved to the US and started working at the office i did.
If you know anyone else that's worked for this company, you'd know. If you haven't used one of their sites today, you probably will this week.
sibling and other comments note that amazon has "consistent look and feel and management and..." so maybe all that "busywork" takes 6 full times?
[^1] i don't count the DBA team as KTLO since i can only think of a single instance when i had to be on a conference and they also were. hurricane in Va!
> 2.3 SDE could do that, but i don't get 6. a 24/7 operation needs 2FTE and 1 PTE per shift, and there are three shifts.
having the team constantly on call is a recipe for having them quit. sure, you could run this with the bare minimum number of engineers, but your turnover would be so high and given how high hiring costs tend to be, this is a net negative
> so maybe all that "busywork" takes 6 full times?
ktlo in a constantly changing company is not easy. software / host patching to maintain compliance is necessary busywork that requires a bit of babysitting. not to mention keeping up with required migrations to new stuff due to internal deprecations.
From all what I've heard over the years, they have a problem staffing the service teams. I imagine with the AI craze that just got worse, as they certainly shifted resources to work on that.
If they really have staffing problems, it'd make sense in the short-term to shut down services which don't bring in much revenue. In the long-term that might fire back if AWS customers loose trust in the services they're using still being there in the future.
The obvious solution to a possible staffing problem would of course be not fire a certain percentage of your employees every year.
I have a lot of grips with AWS, but long term support isn’t one of them.
They’ve given nearly 2 years notice here (more than double what a lot of other SaaS providers offer) and there will be an army of AWS engineers available to help customers migrate too.
AWS do actually discontinue services all the time. It just doesn’t make the headlines because their depreciation process is so good.
Now, if you wanted to discuss their billing, difficultly to navigating the literally thousands of products they offer, contradicting documentation, half baked implementations, cryptic error messages (assuming you’re even lucky enough to get an error) when deployments fail, or the dozen other hurdles that make using AWS a highly paid specialty, then I’d agree with you. But depreciation of services is one of their strengths.
> In the long-term that might fire back if AWS customers loose trust in the services they're using still being there in the future.
Where are they going to go to get higher guarantees of services not being abandoned? Google?
> The obvious solution to a possible staffing problem would of course be not fire a certain percentage of your employees every year.
Have you worked for a BigTech company and seen their promo doc? No developer would want to work on a dead end service who was interested in their career.
Really? I've heard Amazon is so understanding of the difficulty in new hires picking up legacy code, even with the extremely diligent help of the other "mates" on their "team".
I'm sure with the excellent work life balance, supportive atmosphere, nurturing organization, and upbeat attitude, Amazon's staffing issues are just a short blip.
One thing Amazon is known for us a good place for a long term stable career!
It's a good thing that organizations employment is so stable with so many foundational services that so much of our economy depends upon.
For them it's 0.00...1% of their business. For a small customer who deployed on their service it might be 30% or more of their development budget. Of course that's the way corporations do business. But that's why as a small one you can't really trust any of them. You can just place bets and sometimes you lose.
To put it another way, it is an implicit cost of using AWS: You have to have contingency plans to cope with discontinued services, and you need to be able to realistically execute those plans. I think it is just another reason that AWS isn't as good a deal as it might initially appear to be for small businesses. (Larger organizations may have a better chance of amortizing these costs over many projects.)
Feels like Amazon are turning into Microsoft slowly. Try everything, spread themselves too thin and then shitcan it later. I am reluctant to use anything other than their core IaaS and EKS stuff. Some of the ancillary stuff appears to be maintained by two drunk guys in a shed somewhere as well.
Yes, there is a long migration period (almost 2 years). Yes, if you are paying for support, they will help you a lot with the migration.
But if you are using a (costly) managed service it's because you don't want the hurdle of managing the service. Migrating off something is instead probably the most time consuming thing in a service lifecycle. So, even if it is normal to sunset products and people deal with it all the time, it's still a big PITA if you are affected.
> But if you are using a (costly) managed service it's because you don't want the hurdle of managing the service.
On the other hand part of the trade offs of using such services is that you have no control of when they will become unavailable. Its not just normal, its expected.
I doubt very many businesses have anything but a surface level "vaguely gestures toward Azure maybe" plan for if EC2 ever shuts down.
I don’t have a plan for what happens if TCP goes away either, but that also seems fine.
EC2 and S3 are backbone/foundational enough that I expect they’ll be running the last year that AWS exists.
Most SMEs do not even seem to have backups with another provider.
I cannot see EC2 being shut down but other services might be, and there are all sorts of other issues. The idea that a cloud provider will just take care of everything is delusional. You still have work to do. It might be less work, but its there.
There was an Australian pension provider that recently had an issue with their main cloud account and services shut down. Luckily the regulator required they had backups with a different provider so they were able to restore everything. This is a business running something like AUD 125bn of other people's money.
https://www.theregister.com/2024/05/09/unisuper_google_cloud...
it's not just managed service where you can move to your self hosted version when they stop managing it, it's a fully proprietary stack that's just no longer available to use.
The one service that is not dead yet. But is smelling funny is CodePipeline. It is so bad and they’ve already discontinued CodeCommit (the very bad hosted git service), Cloud9 (hosted IDE that is based on VSCode even though they would never admit it externally) and CodeStar.
A couple comments:
AFAIK cloud9 was never vscode based, it was a dutch startup acquisition when hosted RDEs were looking popular https://siliconcanals.com/amazon-acquires-cloud9/. I believe theyre busy working on similar-but-not-obviously related remote environments like AWS Workspaces.
Re: existing Code-asterisk _services_ see CodeCatalyst at a recent (in progress) attempt to move to a more unified user-centric _product_.
And for downthread Cognito, Id be surprised if it ever goes away. i think theres some unification/deconfliction of capabilities that needs to happen with Federate. But Id expect one or both of those to stick around as a proxy pool of user identities for apps. The more interesting, and huge change to how AWS fundamentally works, is Identity Center(IdC). Tying back to cognito and CodeCatalyst IdC should _finally_ bring user identities, and your corp IdP, in to AWS and actually be useful across existing services. This _could_ catch AWS up to goog cloud and OCI and (I assume) azure on identity management.
Edit: ex AWS, but all if the above based on publicly available information.
Cognito feels like its bones are grinding together
They can’t really get rid of Cognito. It’s the authentication/authorization platform for external users. There is no migration path that will seamlessly transfer user names and passwords to other mainstream services like CodeCommit.
I can see reduced federation options and increased costs, but Cognito is OAuth so it's never going to die. You might as well try to get rid of IAM.
especially since they won't support exporting password hashes.
If you run internal apps on ECS, cognito effectively gives you beyondcorp for free. Being able to put cognito as a rule in your load balancer is choice. I wouldn't use it for anything else, though.
A company with Amazons resources could keep these fringe services around as long as there are paying customers. Just increase the price of the service to match the costs of keeping the legacy service around.
Three problems with this approach. 1. The ongoing “ktlo” (keep the lights on) maintenance cost for a minimal public aws service is on the order of 6 SDEs. Theres a constant stream of security and integration improvements like new IAM policy features, KMS handling, billing & metering, console chrome, docs, etc. These are generally nonnegotiable to keep _any_ sort of consistency across AWS.
This is discounting most operations/operational support, as you cant practically scale on oncall rota that small and your example is probably referring to a small customer base and a very infrastructure light service. You _could_ move that to a cheaper dev center & more shared ops, but that in itself is a lot of work.
2. Theres a lot of overlap between these services. Its often not complete, and theyll have different approaches. But fundamentally Amazons distributed nature means multiple orgs will solve similar problems that are adjacent but outside their focus, and then realize that “product” is someones elses better owned feature. Culling needs tk happen at that user experience level sooner or later. The linked blog post is a good example.
3. “Pay the average cost” is going to be shocking to see how much some of these cost to run. This cost increase is going to feel like extortion for customers who _already use other service_ and probably pay a lot more. Earning ongoing negative feedback over a minor service is not worth it. Nearly all services are priced based on a multi year forward looking marginal cost model. I would very much expect the average cost to be 10x the customer price for a small service like app mesh that never “reached scale” or got their P&L together. Then add in those dev salaries I mentioned.
Most importantly those ktlo dev hours in point #1 are an opportunity cost for AWS. They should be spent on new revenue generating work, not cost reduction, and certajnly not a dead end. More than 10 years ago the rough rule of thumb was that a dev year of work should be targeting $12M/yr in revenue generation. I know we didnt consistently hit that, but thats the kind of benchmark that would be used to evaluate staffing. In AWS today if the overall product doesnt have a line of sight to $500M its not a viable innestment.
You’re forgetting one important point. No one who has any interest in their career inside or outside of Amazon wants to be on a service team for a dead service. That would look horrible on both their promo doc internally and when trying to get through a behavioral interview externally.
Source: former AWS employee
explaining that KTLO requires 6 SDEs is something I wish every executive knew.
It definitely scales of complexity/surface area, and Id say that straight number is a lower bounds fixed cost that you might see from an unsuccessful “small” service. In larger orgs its better a percentage, because its relative to overall investment/growth. My personal goal would be 20% ktlo, usually much closer to 40% in practice. All based on my personal experience doing the cloud thing across 2 companies for 15 years or so.
And I will give credit to AMZN/AWS, of old at least, that the above would be common knowledge with an expectation of much more org specific details for anyone sr manager/PE through VP whod be involved in roadmaps, HC allocation, etc.
are we assuming only 9-5? 2.3 SDE could do that, but i don't get 6. a 24/7 operation needs 2FTE and 1 PTE per shift, and there are three shifts.
And i suppose "6 SDEs" for a service that has... what number of MAU or whatever?
me and a great friend of mine split our roles with our "projects", he deals with paid clients, and i deal with our required services (email, matrix, nextcloud, pastebins, misskey, PBXen, wireguard/VPN/Proxying...) and there's just two of us. I can't even think of the last time we had downtime, except at the tail-end of last year, one or the other of us caused a fault in proxmox, where commands were no longer issuing, and we had to do a reboot and fsck, the drill.
I understand at "AWS" scale obviously there's going to be more fires than 1 or two people can put out, but 6 SDEs is over 2 million in compensation all told for amazon, right?
So i guess what i am asking for is context. I've run stuff with hundreds of millions of monthly unique (for a paycheck). That company had over a billion uniques across all of their verticals 14 years ago. One network admin (like senior, but still). When i was there, there were 4 DBAs. there was 1 who could deal with developers, the oracle/postgresql guy who is one of my favorite humans ever - he could convert C-level grunts into sql queries or whatever it is they do. Then there was a part time mariadb; and then a full time Oracle DBA for just 1 site. I'm really straining to remember who else was basically infra/ops and it was less than 6[^1] of us for the whole company. We did have a NOC in another country but prior to my arrival all they were really trained to know how to do was page the correct person in the US. Before i left, 3 of the NOC people moved to the US and started working at the office i did.
If you know anyone else that's worked for this company, you'd know. If you haven't used one of their sites today, you probably will this week.
sibling and other comments note that amazon has "consistent look and feel and management and..." so maybe all that "busywork" takes 6 full times?
[^1] i don't count the DBA team as KTLO since i can only think of a single instance when i had to be on a conference and they also were. hurricane in Va!
> 2.3 SDE could do that, but i don't get 6. a 24/7 operation needs 2FTE and 1 PTE per shift, and there are three shifts.
having the team constantly on call is a recipe for having them quit. sure, you could run this with the bare minimum number of engineers, but your turnover would be so high and given how high hiring costs tend to be, this is a net negative
> so maybe all that "busywork" takes 6 full times?
ktlo in a constantly changing company is not easy. software / host patching to maintain compliance is necessary busywork that requires a bit of babysitting. not to mention keeping up with required migrations to new stuff due to internal deprecations.
From all what I've heard over the years, they have a problem staffing the service teams. I imagine with the AI craze that just got worse, as they certainly shifted resources to work on that.
If they really have staffing problems, it'd make sense in the short-term to shut down services which don't bring in much revenue. In the long-term that might fire back if AWS customers loose trust in the services they're using still being there in the future.
The obvious solution to a possible staffing problem would of course be not fire a certain percentage of your employees every year.
I have a lot of grips with AWS, but long term support isn’t one of them.
They’ve given nearly 2 years notice here (more than double what a lot of other SaaS providers offer) and there will be an army of AWS engineers available to help customers migrate too.
AWS do actually discontinue services all the time. It just doesn’t make the headlines because their depreciation process is so good.
Now, if you wanted to discuss their billing, difficultly to navigating the literally thousands of products they offer, contradicting documentation, half baked implementations, cryptic error messages (assuming you’re even lucky enough to get an error) when deployments fail, or the dozen other hurdles that make using AWS a highly paid specialty, then I’d agree with you. But depreciation of services is one of their strengths.
Nit (since you typed it wrong twice): it's deprecation not depreciation
> In the long-term that might fire back if AWS customers loose trust in the services they're using still being there in the future.
Where are they going to go to get higher guarantees of services not being abandoned? Google?
> The obvious solution to a possible staffing problem would of course be not fire a certain percentage of your employees every year.
Have you worked for a BigTech company and seen their promo doc? No developer would want to work on a dead end service who was interested in their career.
Really? I've heard Amazon is so understanding of the difficulty in new hires picking up legacy code, even with the extremely diligent help of the other "mates" on their "team".
I'm sure with the excellent work life balance, supportive atmosphere, nurturing organization, and upbeat attitude, Amazon's staffing issues are just a short blip.
One thing Amazon is known for us a good place for a long term stable career!
It's a good thing that organizations employment is so stable with so many foundational services that so much of our economy depends upon.
I mean this is a 20-month notice, that's already fairly generous by modern standards.
It's an opportunity cost. Since they can't have infinite employees, their current employees are better allocated to other things.
For them it's 0.00...1% of their business. For a small customer who deployed on their service it might be 30% or more of their development budget. Of course that's the way corporations do business. But that's why as a small one you can't really trust any of them. You can just place bets and sometimes you lose.
To put it another way, it is an implicit cost of using AWS: You have to have contingency plans to cope with discontinued services, and you need to be able to realistically execute those plans. I think it is just another reason that AWS isn't as good a deal as it might initially appear to be for small businesses. (Larger organizations may have a better chance of amortizing these costs over many projects.)
Better start keeping an eye on this page: https://github.com/SummitRoute/aws_breaking_changes
Feels like Amazon are turning into Microsoft slowly. Try everything, spread themselves too thin and then shitcan it later. I am reluctant to use anything other than their core IaaS and EKS stuff. Some of the ancillary stuff appears to be maintained by two drunk guys in a shed somewhere as well.
I think the economics drive the same way for all companies.
This is old news dated 24 SEP 2024.
Oh this is just moving to Envoy as the reverse proxy.