The 'I work in I.T' thread

charltjr · Jul 19, 2024

Many people at Crowdstrike got that creeping ice water down the spine sensation really badly last night as the true horror of the issue dawned.

Can you imagine the shitstorm!

Advikaz · Jul 19, 2024

Rojer said:
No one has the money for pre prod and testing mate. Just send it, be reet!

DevOps initttt

Krarl · Jul 19, 2024

This is a massive longshot

But does anyone know of anyone selling any Supermicro Microcloud Epyc chassis and blades? Just been let down by a supplier for a deadline next week and these were meant to arrive today, bunch of c***s

Memory isn't too much of a sticking point as some of the customers need 128GB

KitsonRis · Jul 19, 2024

Advikaz said:
Always deploy major changes straight to prod and do it as close to the weekend as possible.

This is our rule at work! No Ops deployments on a Friday unless you realllly have to to fix something.

Krarl · Jul 19, 2024

KitsonRis said:
This is our rule at work! No Ops deployments on a Friday unless you realllly have to to fix something.

Nothing past Wednesday should be the rule for dev/networks changes

Oh and stop testing in prod. Bunch of clowns🤣

KitsonRis · Jul 19, 2024

Krarl said:
Nothing past Wednesday should be the rule for dev/networks changes

Oh and stop testing in prod. Bunch of clowns🤣

Ops testing is fun though….haha.

Or what I’ve seen is people do code, get it reviewed, I’ve deployed it to a test environment and a smoke test, doesn’t work then I debug it to an error where it would never have compiled so how that dev decided “yep that will do” without even running their code baffled me.

.Joe · Jul 19, 2024

Graphics team have just come back with a draft redesign for advertising

charltjr · Jul 19, 2024

I get to run some recursive delete commands on one of our EC2 instances later today. As root. What could go wrong.

Maccy · Jul 19, 2024

charltjr said:
I get to run some recursive delete commands on one of our EC2 instances later today. As root. What could go wrong.

View attachment 1702560

Hope you use `sudo` to really mean it

rctempire · Jul 19, 2024

Donny_Dog said:
Microsoft+Crowdstrike f**k up combo it appears.

I'm flying later. f**king w*****s.
I want to see the RFC presented for this change at CAB.

We are doing the change Friday avvro mate. No worries. Have a cold one and enjoy the weekend.

rctempire · Jul 19, 2024

What a day, it really made my day. My mate calling me saying his estate is in tatters and I said he was having a CrowdStroke.

We laughed but I think he's crying.

.Joe · Jul 19, 2024

I'm very thankful for not having to deal with end users.
but i'm very thankful that 1000+ users are aged enough to understand a guide to using DOS

hopgop1 · Jul 19, 2024

I feel bad for all the first line guys who've got to try and talk someone through the workaround remotely, all on a Friday when the user will definitely be cooperative as they're so desperate to work on a Friday.

GS1qZZlWIAAk5NC?format=jpg&name=large.jpg

charltjr · Jul 19, 2024

Maccy said:
Hope you use `sudo` to really mean it

Every day is xkcd day, but especially gone 5pm on a Friday when you're still working

Cookie · Jul 19, 2024

KitsonRis said:
This is our rule at work! No Ops deployments on a Friday unless you realllly have to to fix something.

I tried to get a customer to redeploy a pre-prod K8s cluster because we (the vendor) were trying to troubleshoot something for them.. at 1650 this evening 😂

(He told me it would have to wait til Monday, spoilsport)

rctempire · Aug 1, 2024

Question. What you lot using to update applications on machines as a whole? Via intune or script would be smashing.

We have used PatchMyPC in the past, but not seen anything of late that seems to be the better one.

andybond · Aug 1, 2024

rctempire said:
Question. What you lot using to update applications on machines as a whole? Via intune or script would be smashing.

We have used PatchMyPC in the past, but not seen anything of late that seems to be the better one.

Qualys.

However SCCM is 99% as good

rctempire · Aug 1, 2024

andybond said:
Qualys.

However SCCM is 99% as good

SCCM is being dumped. Hence the full move to intune. Customer was 100% no server infrastructure. Only "cloud."

andybond · Aug 1, 2024

rctempire said:
SCCM is being dumped. Hence the full move to intune. Customer was 100% no server infrastructure. Only "cloud."

rctempire · Aug 1, 2024

andybond said:
View attachment 1704709

Yep, words out their top IT people.
"Nothing in house, all cloud."

Ok enjoy spending more money forever! Finance bean counters will love it.

andybond · Aug 1, 2024

rctempire said:
Yep, words out their top IT people.
"Nothing in house, all cloud."

Ok enjoy spending more money forever! Finance bean counters will love it.

I get “the cloud” on some levels. Files , email et al.

The whole put it all in the cloud mantra just isn’t right though. Lift a VM and put it in AWS as another instance just isn’t correct in my opinion.

However, as a contractor I am looking forward to the rain ( when people bring things out of the cloud back to on prem )

.Joe · Aug 1, 2024

rctempire said:
Question. What you lot using to update applications on machines as a whole? Via intune or script would be smashing.

We have used PatchMyPC in the past, but not seen anything of late that seems to be the better one.

Depending on Sizing we use 2

Sub 100 endpoints we use Action1
Anything larger we're using Atera together with intune for its updating and reporting

.Joe · Aug 1, 2024

If you're lazy you could set the following to kick off once a month from task manager

"winget upgrade --all"

Krarl · Aug 1, 2024

andybond said:
I get “the cloud” on some levels. Files , email et al.

The whole put it all in the cloud mantra just isn’t right though. Lift a VM and put it in AWS as another instance just isn’t correct in my opinion.

However, as a contractor I am looking forward to the rain ( when people bring things out of the cloud back to on prem )

I'm loving it too because people have seen sense and are now starting to fill out my Datacentre and I'm charging them a f**king fortune🤣🤣

andybond · Aug 1, 2024

Krarl said:
I'm loving it too because people have seen sense and are now starting to fill out my Datacentre and I'm charging them a f**king fortune🤣🤣

Which company?

charltjr · Aug 1, 2024

Cloud is amazing, with the right use case. You want to scale on a whim, with workload, and might have to deal with some massive peaks from time to time? f**king excellent, fill your boots.

Just dumping a VM from an in-house server to a cloud server, "because cloud" and you want to fire your IT team.....

rctempire · Aug 1, 2024

.Joe said:
If you're lazy you could set the following to kick off once a month from task manager

"winget upgrade --all"

View attachment 1704718

Essentially the manual way of WinGet-AutoUpdate which does this in a semi automatic fashion. Not doing this for thousand endpoints. Plus customer needs reports for CE compliance.

Krarl · Aug 1, 2024

andybond said:
Which company?

I'll pm you if you're after some Colo?

Krarl · Aug 1, 2024

charltjr said:
Cloud is amazing, with the right use case. You want to scale on a whim, with workload, and might have to deal with some massive peaks from time to time? f**king excellent, fill your boots.

Just dumping a VM from an in-house server to a cloud server, "because cloud" and you want to fire your IT team.....

View attachment 1704733

Until your 2 years of favourable pricing is up and then they introduce you to your new and improved 100% markup contract that you're now struggling to get out of and you're being dry humped over🤣

charltjr · Aug 1, 2024

Krarl said:
Until your 2 years of favourable pricing is up and then they introduce you to your new and improved 100% markup contract that you're now struggling to get out of and you're being dry humped over🤣

That would be most unethical.

Excellent.

.Joe · Aug 1, 2024

Krarl said:
Until your 2 years of favourable pricing is up and then they introduce you to your new and improved 100% markup contract that you're now struggling to get out of and you're being dry humped over🤣

Does your email signature contain broadcom somewhere in there? 👀

Donny_Dog · Aug 1, 2024

charltjr said:
Cloud is amazing, with the right use case. You want to scale on a whim, with workload, and might have to deal with some massive peaks from time to time? f**king excellent, fill your boots.

Just dumping a VM from an in-house server to a cloud server, "because cloud" and you want to fire your IT team.....

View attachment 1704733

I love having presales conversations about cloud.
It's nuts to think that folk think it is a replacement for on-prem. Some think it's like for like. They have no clue of the cost model, tie in, where cloud is actually good, etc.
Mental. Just on the hype train.

Advikaz · Aug 1, 2024

I move a lot of clients to cloud platforms.

Makes a lot of sense for SOME firms. Depends how they work, it's definitely not the one single answer to every network problem like it's often portrayed, as we know.

The amount of business's out there paying shitloads for hosted servers for effectively what is, a DC and their file repositories. It's crazy when you can just bin the lot and bung the file data in SP or similar which is usually in their 365 licencing anyway, and AAD/Entra them.

charltjr · Aug 1, 2024

At our place we couldn't sell our cloud SaaS solution for years, almost everyone wanted on-prem. Suddenly it was like someone threw a switch and now all we can sell is SaaS.

Same product, same solution, we've gone from six cloud customers two years ago to over twenty. It is literally just that the finance and banking sector have suddenly pivoted to cloud, even the few customers who are "on prem" are deploying it in their own private cloud. Very few of our customers have actual physical servers any more, the transition has been incredibly fast.

Which is all well and good, until Azure B2C sharts itself, or AWS pull part of the infrastructure out from under us, or you actually take a close look at those hosting costs...... Always on, fixed size instances. Yeah, not exactly the best use case for cloud.

Krarl · Aug 1, 2024

Theres no such thing as the f**king cloud though. People talk about it like it's some sort of Jesus like creature and packets whizz above our heads like in Charlie and the chocolate factory

It's just in someone else's tin in someone else's Datacentre that they're absolutely rinsing you for.

Apart from that last bit, we don't rinse you for it. Pinky promise?

andybond · Aug 1, 2024

Krarl said:
Theres no such thing as the f**king cloud though. People talk about it like it's some sort of Jesus like creature and packets whizz above our heads like in Charlie and the chocolate factory

It's just in someone else's tin in someone else's Datacentre that they're absolutely rinsing you for.

Apart from that last bit, we don't rinse you for it. Pinky promise?

There are 3 states really

Cloud - see f**k all but you know it’s there and does what it wants.

Fog - your kit but you can’t see it

Correct - what you should have unless it’s SaaS

ChrisR · Aug 1, 2024

I was dealing with a client a few years ago who had our on prem product.

They needed to move over to a SaaS solution as the company had exited/sold all their data centres globally.

This was a £120 billion company, I wasn’t complaining too much as sold them $2mil of stuff to get them in the same position they were in before 😂

(Got screwed on the commission for that one which contributed to me leaving not long after, but that’s another disaster story)

Krarl · Aug 1, 2024

Sounds a lot like my old employer which ryhmes with NoBaddy

They've shrunk all their DC infra, sold us off and now have zero presence in the UK for hosting (pops don't count). They came over here like the wild west expecting to take the UK market by storm... And flopped tremendously because the higher ups are ALL yanks that have zero f**king clue

I wouldn't trust the c***s to run a bath nevermind a business. Atrocious. Only good thing about them is they pay alright and there's incentive to stay with the amount of shares they give

Maccy · Aug 2, 2024

Government entity I work for are all hosted on Azure, in fact I think most of the gov IT is hosted on Azure, but we're in the process of introducing HPA across everything and it's forecast to save a few hundred k per year.

I don't claim to know much, I just test the apps, but find it interesting how the costs can really snowball.

boultonn · Aug 2, 2024

Maccy said:
Government entity I work for are all hosted on Azure, in fact I think most of the gov IT is hosted on Azure, but we're in the process of introducing HPA across everything and it's forecast to save a few hundred k per year.

I don't claim to know much, I just test the apps, but find it interesting how the costs can really snowball.

The dept I'm currently contracting for is both Azure and AWS, primarily AWS because of all the service offerings.
Dread to think what the total spend is, but I will say that they make use of cloud benefits, so using native services like Margate to reduce patching burden and auto scaling groups for scaling and cost saving.

The 'I work in I.T' thread

Similar threads