Because storing PBS archives is a voluntary public service, and storing other random peoples' stuff isn't.
ares623 16 hours ago [-]
What a wild idea
usrnm 13 hours ago [-]
> indefinitely
Which, in practical terms, means "while I or my investors have money". These guys have 70 years worth of data, can you guarantee to keep it for another 70? The real answer to these kinds of problems is managing the data yourself
AdamN 11 hours ago [-]
There's a difference between an archive and a backup it's true. If this really is intended to be archival they need to come up with a plan with the Library of Congress, the US Archives, or a similar institution. There are archivists in the world and they should come up with a plan for this that's written and will outlast all the people in the room.
itsamario 6 hours ago [-]
I'm surprised libraries haven't taken a similar route to universities and store local data and backups for neighboring libraries.
Libraries were the archives for newspapers and local public information.
krapp 5 hours ago [-]
I doubt many libraries have the budget for it. Libraries aren't a popular service and are perennial targets for conservative "cost cutting."
adolph 6 hours ago [-]
Not certain how well LOCKSS has done over time as Internet Archive seems to get all the press, but it seemed to be a solid concept with some level of implmentation:
Taking its name (Lots of Copies Keeps Stuff Safe) from the idea that a file is more likely to survive if it has multiple copies stored in multiple places, LOCKSS software allows libraries to create preserved digital collections out of materials that would otherwise be accessible only through a licensed academic subscription. LOCKSS has agreements with academic publishers to permit libraries to store local copies of content that is usually only accessible through web-based subscriptions, enabling post-cancellation content access.
At present (especially for PBS), it would be wise to use an archive service that receives zero U.S. federal dollars and is not subject to the executive branch’s efforts to rewrite history to fit their preferred narrative.
tim333 4 hours ago [-]
I'm thinking of ways to do that. I think maybe partnering with long lived institutions like universities or national libraries?
gaze 6 hours ago [-]
rsync.net is a special exception. Maybe the most reliable and trustworthy service money can buy.
I highly suggest (if you haven't already done so) you contact Nine PBS, as they (Nine PBS) won't be reading Hacker News.
Barbing 15 hours ago [-]
(random user here) Emailed Nine PBS
Edit per “no contact on profile” comment:
Contact info@rsync.net for more information, and answers to your questions.
throwaway270925 11 hours ago [-]
Out of general interest: whats your bus factor?
skoskie 1 days ago [-]
Epic.
1 days ago [-]
remuskaos 1 days ago [-]
fucking legend
jeremyjh 22 hours ago [-]
It’s entirely possible that Iron Mountain will be happy to turnover the data but needs a court judgement to do so without incurring more legal exposure. The article says the system belongs to OSS. If this is something like a colo arrangement then probably that is the only reasonable position to take. They can’t just go crack open a customer’s server and pull data out of it for whoever asserts a claim on it.
to11mtm 20 hours ago [-]
Iron Mountain is typically very stringent on their retention/disposal policies.
Storytime, since I'm no longer under NDAs for the company, over a decade ago somewhere that I worked, a disgruntled employee who decided to quit, made a point to within a week of quitting, take a whole bunch of important physical documents they had (because, FBOW, our email and other data retention was so bad, it was safer to have them as prints) and threw them all in our fancy Iron Mountain 'disposal' bin.
Apparently due to the agreement with them, short of a court order, only specific employees/titles in the company were allowed to be the persons to retrieve them.
Keeping that situation in mind, it's clear that this dance for better or worse is part of their posturing as a retention center with very strict rules to ensure compliance while also avoiding snooping.
MadnessASAP 17 hours ago [-]
As a whole, I'd say it's for better. Obviously Nine PBS's situation is shitty and fuck "OSS" for putting them in that position, but they are not Iron Mountains customer. They are strangers to Iron Mountain who just demanded 50TB of customers data from Iron Mountain. The only sane response is to tell Nine PBS to come back with a court order, and (hopefully) to tell their lawyers to just make sure Nine PBS is in fact the rightful owner and if so, not to fight them on it.
Anything else is the sort of stuff that causes blog posts and news articles about data breaches to be written.
namibj 11 hours ago [-]
Yeah; I haven't read TFA but IMO especially if Iron Mountain is aiding the procurement of said court order through giving some (potentially redacted as required for privacy purposes) "sample" court documents that could be readily used as reference so the lawyer on the Nine PBS side (filing with the court to get a judge to order Iron Mountain to release that data to Nine PBS) doesn't need to waste their time repeating all the drafting from scratch for what I'm assuming would be a (from the legal perspective) straight-forward request (that nonetheless has to specifically account for the particular legal data possession/ownership/access-rights situation under which the data in question is stored at Iron Mountain), especially as relevant for the situation at hand allowing Nine PBS to put a temporary data retention legal hold on these 50TB giving them a fair deadline (not merely "legally 'reasonable'") either on charity (understanding the gravity of the "this Nine PBS data is a public good (at a cursory glance)"/"information wants to be free") or offering general reasonable list price of what they bill for retaining 50TB stored under the storage model at play for "just one more month" (or whatever fair deadline) paid by Nine PBS.
sargun 6 hours ago [-]
I accidentally dropped something into an iron mountain bin years ago, and just pried it open with a screwdriver.
trollbridge 18 hours ago [-]
Er... I assume a little time with a lockpick and/or an angle grinder could have recovered the documents?
alephnerd 17 hours ago [-]
Hope you have (best case) a good lawyer or (worst case) good health insurance as well if you attempt physical intrusion. Their physical asset security team is hardboiled [0][1].
I've dealt with them as well - they don't like being in this kind of position either.
If Iron Mountain is shooting people for picking locks, they’re the ones who had better have good lawyers.
gruez 7 hours ago [-]
Right, but if they show up and try to stop you, your options for escalation are limited and you'll have to comply (stop).
adaml_623 14 hours ago [-]
I think you've misunderstood. I believe the comment you're responding to was talking about opening a secure bin that was presumably still on site.
They weren't talking about breaking into the Iron Mountain facility
LogicFailsMe 8 hours ago [-]
Angle grinders are amazing tools, but with such amazing tools comes amazing responsibility... Yes, that's an allegory...
duttish 15 hours ago [-]
I smell a Jason Statham movie
alephnerd 15 hours ago [-]
I smell a felony case with a non-trivial chance of lead poisoning or a visit to a trauma center.
mc32 9 hours ago [-]
Indeed. That said it’s not that different from say someone backing up to someone like Druva and they ran their infra on AWS and say hypothetically Druva went out of business you would not go to AWS directly and say “hand me over my data.” It’d have to go through proceedings, etc., not to mention that without the engineers from Druva you would not get useable unencrypted data.
It’s exactly like the above but it’s presented as if the data sits neatly on two drives stored in a safe and all you have to do is mount them to see the unencrypted data.
medellin 18 hours ago [-]
[flagged]
greazy 18 hours ago [-]
Oddly aggressive comment on something so mundane. Guess it's a pet peeve of yours?
dghlsakjg 20 hours ago [-]
They have a judgment against OSS ordering the data to be returned, and affirming ownership.
mixdup 20 hours ago [-]
That is not the same thing as a judgement ordering Iron Mountain specifically to release the data
doctorpangloss 16 hours ago [-]
Maybe it isn't the same thing in the Iron Mountain Cinematic Universe.
Consider this: the utility that provides the electricity and the ISP providing internet also do not have a direct relationship with Nine PBS, but will be essential in carrying out the court order. Does the judge have to expressly name every transitive party responsible for carrying out the order? Time will tell, but to me the answer seems "no."
jrochkind1 12 hours ago [-]
it sounds like Iron Mountain specifically markets itself as "we will never turn over your data without a court order", so whether contractually obligated to do that or not, choosing to do that when they have a legal choice to do so is part of what attracts and keeps customers, and not choosing to do voluntarily would harm their business and lose them customers. Perhaps they could legally choose to turn over the data, but they choosing not to. A court order would give them no legal choice, and they would then turn over the data, and it would not harm their attraction or retention of customers, because that was always the deal, only with a court order.
mixdup 2 hours ago [-]
Those two entities don't need to do something that they wouldn't normally do for that order to get carried out. It's an extraordinary act for a colocation facility to release hardware or data to someone they have no relationship with. Iron Mountain has no relationship to or with Nine PBS, only with OSS
I'm sure Iron Mountain will be happy to turn over the data, but to cover their ass in terms of their agreement with OSS, they need the cover of a court order otherwise they would likely be in breach of their contract with OSS
everforward 5 hours ago [-]
The electricity company doesn’t have a contract that says they’ll only give electricity to specific people.
Iron Mountain just doesn’t want to be in the middle, and I can see why. OSS went defunct, PBS files a lawsuit, guy buys OSS so PBS pauses, guy now claims he was defrauded into buying it, and somewhere in the middle nobody from OSS showed up in court so there’s a default judgement to turn those over.
I would probably also refuse to turn over the documents without a court order. The whole situation is a mess and the only way to be absolutely sure you aren’t going to get dragged into it is a court order for specific performance.
PBS probably isn’t going to sue them for releasing the docs, but who knows what OSS will do. It doesn’t sound like anyone even really knows who owns OSS at the moment.
namibj 11 hours ago [-]
Pretty sure a court order to return say a "PlayStation Dev kit" a game studio employee had when he got fired during covid remote work mandates, if it's just against the employee personally, would not obligate me to let you in or carry it out to you if that employee left it on the middle shelf of my couch table having told me he'll be back after, say, taking unpaid and non-granted-despite-requested time off for his "common law wife if that was a thing in the state they're in, but legally just unrelated consenting adults (that haven't yet moved in together)" 's funeral and rounding it up to one whole week to grieve at their memorable places, after having crashed at my place (bringing his work with him, thus the dev kit on my table-shelf) as I live much closer to the hospital his "wife" was in for the past 3 weeks and he expressed some psychological worries about spending almost all of his time alone while the love of his life is dying....
In fact, I'd expect you to need a seizure-warrant/distinctly-named-but-to-the-layperson-identical-court-order for my place as, among other things, I don't think I am in a position to legally have standing to challenge an order that's not mentioning me, and lack of me being an active flight risk would probably preclude you from arguing exigent circumstances to secure the dev kit without a judge ordering me to let you.
TL;DR: "complying" with a court order that's filed against a 3rd party is, IMO/AFAIK, a voluntary act, specifically because the party not mentioned in the court order AFAIK has no legal standing to challenge that very order of in court.
_Precisely_ because they're not legally mandated to comply, at least not beyond perhaps taking a precautionary legal hold on the referenced information to not get accused of aiding in destruction of data in case the court-ordered party directs the non-mentioned party to delete through the regular customer relationship channels (API? Whatever else?).
doctorpangloss 6 hours ago [-]
You are overthinking this. In your scenario, you will offer to pay a lawyer to write your reply who who will say, "no just comply with the order who cares." You cannot make analogies. People just hold these frou frou promises from Iron Mountain in some higher regard than other meaningless marketing, so they mook for it - a company nobody cares about - on Hacker News.
kulahan 21 hours ago [-]
It still sucks for all involved, and it's a shame there's not a better solution here. It's gotta be so frustrating for the folks at PBS to keep getting hit like this.
pirate787 20 hours ago [-]
Actually it is PBS that bears responsibility here. Very likely the decision by PBS to hire OSS was corrupt (OSS are just a reseller of Iron Mountain) and they made no other provision to backup the files a second way, like AWS Glacier.
pavel_lishin 18 hours ago [-]
> Very likely the decision by PBS to hire OSS was corrupt (OSS are just a reseller of Iron Mountain)
That's making a lot of assumptions. I host some of my backups on what is effectively a reseller of S3; was that decision corrupt?
But, yes, PBS does bear some responsibility here. If you leave your car in a parking lot overnight with the door open and the key in the ignition, the thief is still wrong for taking it, but there's a limit to how much indignity you can have.
alephnerd 17 hours ago [-]
> PBS does bear some responsibility here
PBS is an association of local public broadcasters, most of whom are SEVERELY underfunded.
roysting 14 hours ago [-]
Although you are correct, the statement is still arguably correct.l, even if only accidentally.
As the parent entity, the national PBS has a responsibility to guide and even mandate policies and provide best practice guidelines or even offer support and assistance in burden sharing arrangements.
PBS has long now had major structural challenges due to things like internal corruption and overbearing control by the well funded chapters without commensurate responsibility.
As with many such decentralized membership organizational structures, incentives can often be very misaligned between verticals and laterals.
alephnerd 18 hours ago [-]
> is PBS that bears responsibility here. Very likely the decision by PBS to hire OSS was corrupt
Don't blame corruption for what was most likely extremely strained budgets. Local public access TV stations like KETC (Nine PBS) tend to be severely understaffed and underpaid.
Realistically, they went with OSS because the quote for a local MSP was cheaper than the cost of hiring additional headcount (a $70k salary ends up costing roughly $100k for an employer).
(Source: occasionally dealt with these kinds of accounts back when I was an SE and later PM)
Edit: I'm right - they only have 1 IT/Software employee managing the entire organization which only has a budget of $30M after losing $1.8M in federal funding.
Oof, that is a horrible position to be in with so many dicks on HN. You'd assume SWEs would have sympathy for other SWEs.
Some HNers should have been taught better manners by their parents or should have at least internalized the lessons from Arthur as a kid [0].
If that employee sees this post - I want to say THANK YOU. It's a hard job.
> You'd assume SWEs would have sympathy for other SWEs.
Only if you haven't been reading HN very long!
alephnerd 7 hours ago [-]
I have - this is my throwaway account, and my primary account has been alive since a little after YC moved from Boston.
I hate HN and how it has gotten worse and worse as the years go by. There are a couple basic changes that the mod team could do to help alleviate the degradation but they aren't doing it.
pjc50 12 hours ago [-]
> PBS to hire OSS was corrupt (OSS are just a reseller of Iron Mountain)
You're going to have to unpack the assumption here: are you alleging that there was a kickback in the procurement process?
vel0city 18 hours ago [-]
Note, this wouldn't be PBS the national organization, this would be the local PBS affiliate who would have made this decision. The affiliates are extremely independent.
Crosseye_Jack 1 days ago [-]
Ok, it sucks, losing data always sucks, but what ever happened to the 3-2-1 backup rule? even if your off site backup provider goes tits up, your on site backup is still there.
If you don't want on site, a second off site provider, 50TB on backblaze would be an extra ~$350 per month (prob less if they were to commit to capacity instead of using their PAYG service)
embedding-shape 1 days ago [-]
> but what ever happened to the 3-2-1 backup rule?
I wasn't 100% sure, but the data backs this up[0]; the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.
I can pretty clearly remember hearing about the "3-2-1 backup rule" by name 10 or 15 years ago on the Security Now podcast with Steve Gibson. So it is _at least_ that old, but probably even older. The concept of having multiple backups is not at all new, however. Since I've been involved with computers (~30 years), I have heard folks say things like "two is one, one is none" in reference to backups and data replication in general.
Also, this TV station has not been sending their show archives to Iron Mountain for the past 70 years. Probably what happened was: In 2019, they had an on-site or nearby archive. Then some enterprising IT director came along and convinced management that they would save a bunch of money and time by outsourcing everything to a third party who said they could do it for cheap. Very confident in their genius, they then scrapped their own archive entirely.
Part of me pities them for ending up in this situation, the other part of me believes they got everything they asked for.
wlesieutre 22 hours ago [-]
The rule I always remember hearing is “however many backups you think you have, you have one less”
Which is not as thorough, but still makes the point that you should do more than one backup
pavel_lishin 18 hours ago [-]
Two is one, and one is none.
bothers 18 hours ago [-]
> Probably what happened was:
Probably! The important part is that your pretend story time supports your base desire to blame the victim!
layer8 1 days ago [-]
The concept of multiple copies in multiple locations already existed when backups on CD-R were common, so 20+ years ago. The “2” in the 3-2-1 rule refers to using different physical storage media and predates cloud storage.
olyjohn 1 days ago [-]
This has been a rule for like 30+ years (I have only been in the industry that long). Whether it was called "3-2-1" before that or not, the practice is the same. Have an onsite backup, have an offsite backup. This isn't a new concept at all.
dylan604 22 hours ago [-]
Your description leaves the reader wanting. You make it sound like having a backup on an HDD in one location and then having another copy on the exact same make/model HDD in another location is a-okay. What if the vendor had a defect that strike the make/model of HDD used?
Dylan16807 12 hours ago [-]
That's pretty unlikely, but even if it does happen to be a 3TB Seagate or something you'll be pretty safe. The offsite backup is probably offline so no drives are failing, the onsite backup is still pretty likely to survive, and you have the originals.
dylan604 7 hours ago [-]
You say that to someone that has personally brought up 4 16-drive 4RU units on a Friday with brand new drives and then left over the weekend to complete the striping and an initial warm up process of writing/deleting files to the array. When I arrive on Monday, multiple drives in each enclosure had failed which meant not data recovery would have been possible. Luckily, it was during the warm up and not 2 weeks later after being active. After contacting our vendor, all drives were replaced not just those that failed. Later discovered that it was a manufacturer issue, and multiple other clients of our vendor had similar experiences.
While statistically it might equate to "unlikely" as you state, the fact it did happen, I now have that experience that is not forgotten. Never bring a large storage pool into PROD immediately after the array finishes striping. Put it under load for a period of time first. The question remains on how long that period is though which is probably answered with "how paranoid are you"
Dylan16807 1 hours ago [-]
> Luckily, it was during the warm up and not 2 weeks later after being active.
While there's always luck involved, dying during warmup and not two weeks later is the expected experience for drives that defective.
So what happened to you wasn't that you made a backup and then it failed so fast you couldn't use it. It's relevant to what I was saying is unlikely, but it's not an example of it.
Plus at zero week or even two weeks you have the previous backup system around. If you have a big simultaneous failure 6 months out, that's a scary situation, but that's much rarer.
BeetleB 1 days ago [-]
> the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.
They signed up with them in 2019. Not long ago.
Even if it is a "new" concept, common sense isn't. Pretty much anyone who cares about their data will do the due diligence, and this is not something that requires technical chops. This has been a concern for well over 20 years!
It really sounds like some senior management trying to be cheap.
jrochkind1 11 hours ago [-]
It is definitely not having sufficient budget, although I doubt management was "trying" to be cheap.
They probably have 1-2 IT staff, who have been paid significantly under market value compared to private sector.
Management was not management of an IT organization, and probably did not really understand the trade-offs or what they'd lose by not spending more money on IT. Their IT staff was probably super busy and over-worked and also (not to throw them under the bus) not experienced enough to really advise them, and if they had and had tried to advise them, would management have decided that doing less programmatic work to pay for it, and having IT people at least temporarily spend less time on whatever they do to support regular business, to set up and support backups, was worth it? Maybe, maybe not. Again, the managers probably have little experience with IT. The whole organization is probably well under 100 people.
Source: I work in non-profits.
Why is HN always focused on suggesting there's a person who could have easily made the right choice if they weren't an idiot?
BeetleB 5 hours ago [-]
You say management was not being cheap, and then list everything cheap management does.
I'm familiar with the dynamics of such places.
If they actually go through a proper lawsuit, my guess is the lawyers will cost more than whatever they'd have to pay for redundant backups by an order of magnitude. Which means they probably won't go to court.
supriyo-biswas 1 days ago [-]
> I also remember things like "have one backup in one place, another in another place" since before this mnemonic
I've heard variations of this all the way back to some 1990s vocational IT courses in the country I live, and it wasn't even technologically advanced at the time. It's hard to have any sympathy for someone who misses this concept in today's age.
xtajv 1 days ago [-]
I'm under the impression that "two is one and one is none" is old military slang.
dwattttt 22 hours ago [-]
Redundant backups arranged to survive things like "building burns down" or a natural disaster predate digital backups.
PeterStuer 12 hours ago [-]
Maybe the catchy name, but the practice was very much there in the 1980's.
Crosseye_Jack 1 days ago [-]
The 3-2-1 rule is ~20years old at this point. But just because an archive is "old" doesn't mean we can't apply modern standards to that archive as time marches on, otherwise the BM_ARC would have never been build and opened a couple of years ago.
I mean, that 70+ year old archive must have been digitised at some point, unless Iron Mountain was storing the reel to reel tapes and/or film from the original recordings. So some "modernation" has been applied to it since its creation.
Melatonic 1 days ago [-]
Wouldn't be surprised if it wasn't digitised. TV was using BetacamXR up until very recently (and still might be). Tape is pretty reliable and lasts awhile.
Now add in tons of other legacy formats and digitising becomes a pretty big cost for a public station.
phire 21 hours ago [-]
I’m not sure this is the only copy of the data.
Sounds like they paid a company to digitise all their old tapes, and store the results. They possibly still have all the tapes (on site) and are just trying to avoid paying the costs to redigitise them.
But, it’s also quite likely they destroyed all original the tapes after digitisation, and this was the only copy.
account42 13 hours ago [-]
Tapes decay so it would still be irresponsible to have only one copy of the digitized versions. It's still possible that they have other copies though that are just harder to access.
Tangurena2 9 hours ago [-]
NASA stored all the data from space missions on tape - those old reel to reel mag tapes like you see in old SF movies. Decades of using floor buffers ended up erasing the bottom 2 rows of tapes.
Usually what kills the tape is the little reflective stickers on the back side of the tape material will fall off and render the tape useless since the machinery can't tell where the recording starts. Apparently I bought cheap mag tapes back then.
basch 1 days ago [-]
From the article: It's not clear if the channel has backups of its data, but based on its determination, we suspect not. This is strange, though, especially for a major organization, as one can easily purchase a six-bay NAS on Amazon and populate it with the best HDDs. This also goes against the 3-2-1 backup rule, which says you should keep three copies of your files stored in two different mediums, with one of them running offsite. Hopefully Nine PBS will implement a better backup system once it regains access to its data.
kmeisthax 18 hours ago [-]
A different article about this (which I believe you're quoting) states Nine PBS has about 50TB in limbo, which actually smacks me as kind of small. I had written up an entire screed about the problems with archival[0] and how Hollywood is getting hit with huge costs to maintain LTO tape libraries; because when you get to the PB scale doing this correctly becomes nontrivial and outsourcing to a specialist is genuinely a safer bet. I genuinely have more data to lose than they do.
At 50TB, they probably could have Just™ written three copies of the data to a pile of disks, sent one copy off to Iron Mountain, and kept the other two copies unplugged in a safe. I can understand the DIY solution might not be allowed for insurance reasons, but it's something so cheap they probably could have done both. Unless OSS was just taking them for a ride on storage, which... well, actually that seems pretty likely.
[0] To put it simply, there is no good medium for the archival of digital information. HDDs are not shelf-stable. Flash is even less shelf-stable than disk. Sony is actively trying to kill optical. LTO is shelf-stable, but good luck buying known-good or new drives for old formats. The only option is periodic cloning and verification of offline disks, or keeping all your disks online, which is the NAS solution.
dredmorbius 23 hours ago [-]
The backups (and servers) seem to still exist, but KETC's vendor is estranged with its own hosting / siting provider, Iron Mountain, which is a going concern.
IM seem to be holding KETC's data hostage, which is Not Cool.
scosman 22 hours ago [-]
It's not that simple. If I'm a host and someone shows up without the keys or passwords and says "that's my data, I was a client of a client of yours", I have no idea how to tell if that's true and no idea how to deliver the data.
Maybe they mixed data from many companies on the same volume like S3 does. Maybe there were ACLs on that. Maybe it includes PII. Maybe it's a bunch of shards and needs an index on OSS servers to reassemble. Maybe it's encrypted by keys OSS had.
dredmorbius 21 hours ago [-]
1. If you're hosting third-parties at scale (as in: more than one), you'll have procedures for validating ownership and access.
2. That doesn't actually seem to be Iron Mountain's argument here, though the reporting isn't crystal on the point.
compiler-guy 20 hours ago [-]
For the first level customers, sure. But their customers’ customers? Almost certainly not.
There is no way for Iron Mountain to determine who OSS’s customers are, and their permissions, without help from OSS itself.
And because OSS is defunct, they can’t get that help.
vel0city 18 hours ago [-]
So imagine I decide to make some tool to let my customers store their files. I just decide to toss them all in a big fat S3 bucket behind the scenes with what look like random UUIDs for the keys in the bucket. All the relationships between whose files are whose are in some other database not hosted on AWS.
How does AWS then determine which files are whose when my angry clients go knocking on their doors because I've decided to disappear?
lenerdenator 1 days ago [-]
It's worth noting that PBS affiliates are non-profits that have fairly constrained budgets, especially now that the Corporation for Public Broadcasting has been dissolved due to DOGE.
Non-profit IT has always seemed, to me at least, to be a land of compromises. There's rarely enough money, knowledge, or time to do things "the right way", so you have to do the best you can. If I could only choose one backup to have, it'd be remote on a provider like Iron Mountain. It would seem as if they are storing things on Iron Mountain through an intermediary company, who may or may not have actually paid the bill.
So, y'know, fraud's a possibility here.
cwnyth 22 hours ago [-]
Iron Mountain is actually an incredibly expensive option, and their agreements are predatory. I would not recommend them.
SwellJoe 20 hours ago [-]
Old-fashioned as hell, too. My employer needed to put our stack in escrow for a large deal, and Iron Mountain was where it went. They pass Word docs back and forth like savages.
epistasis 21 hours ago [-]
Their contract was with OSS, a company that went defunct, that had apparently been using Iron Mountain underneath.
cwnyth 19 hours ago [-]
Thanks for the clarification.
quietsegfault 20 hours ago [-]
I used iron mountain for offsite storage of tapes for a huge scientific data archive. The price seemed reasonable given the service we had.
cwnyth 19 hours ago [-]
At a company I worked for, we used them for off-site storage, and when we finally went looking for alternatives, we realized we were being charged thousands of dollars a year more than competitors. We no longer have a contract with Iron Mountain.
Mistletoe 1 days ago [-]
[flagged]
WorldMaker 1 days ago [-]
It's nice to know that even in the end the CPB lived up to its mission and promise and swallowed its poison pill rather than be corrupted by partisan politics, but it's also gross to have seen that forewarned poison pill needed in my lifetime. There's some hope that we can still find the next Mr. Rogers to put a heart on a sleeve in front of some future Congress and build a sequel CPB with similar strong core principles.
ameliaquining 16 hours ago [-]
The U.S. was way less politically polarized back then.
LogicFailsMe 8 hours ago [-]
I think we were just as polarized, if not more so, but we didn't have 24/7 ragebait egging us on to fight some more to drive up the engagement numbers. It's like someone watched the Star Trek ToS episode "Day of The Dove" and thought it was a great idea for a startup. But also, that was 1968, plus sa change and all that...
By their metrics, clearly, they are correct. But reading some books lately that were either from or set during the mid-70s to early 80s, the same tired political tropes were already in the zeitgeist.
But also, from that link...
"Americans' views on gun control seem to stem almost entirely from their cultural worldview and how they position themselves in that cultural schema. Statistics do not persuade them to change their minds."
If true, stick a fork in us, we're cooked.
pirate787 20 hours ago [-]
Sesame Street is doing just fine and the people making billions off the licensing will have to make do without federal taxpayer subsidy.
pirate787 20 hours ago [-]
Congress voted to stop federal funding of the CPB, it had nothing to do with DOGE.
In a free society, we don't need state-run media.
jancsika 5 hours ago [-]
> In a free society, we don't need state-run media.
I think this is a kind of time-sequence fallacy which I've seen before on HN. It's not that "we don't," but that, perhaps, "we wouldn't."
E.g., I study the pieces still standing in famous checkmate positions and decide to start the game using only those pieces. Bad strategy.
rswail 12 hours ago [-]
In an oligarch owned media ecosystem, with obvious corruption of the regulators (FCC, DOJ) to ensure proper use of the public radio spectrum and monopoly restrictions, having a non-partisan publicly funded independent media is vital to the health of democracy.
The US is not a "free society".
Which is why the BBC, ABC, CBC, etc exist in civilized societies.
mindslight 6 hours ago [-]
I think it's worth putting a finer point on the dynamic that the US's real state-run media IS FOX, NBC, and their ilk. The main power structure in the US is the corporate conglomerates, whose owners/managers may compete with one another like players of a sporting team compete with each other, but are ultimately all playing against the People. So invoking "state run media" to talk about PBS/NPR/CPB taking a little public money and trying to produce a different angle of public good rings totally hollow. If you don't like what they end up producing, you can always not tune in! But by the time you're rejecting them on ideological grounds, you should have already rejected every other single station on the air apart from community access type channels.
walrus01 20 hours ago [-]
The news article is really not clear about whether this was, relative to the company "OSS":
a) OSS is a colocation customer with its own hardware colocating it inside an Iron Mountain datacenter. In which OSS owned the bare metal and paid iron mountain for rack space and power.
or
b) OSS is a dedicated hardware customer of Iron Mountain running a service on bare metal owned by Iron Mountain, and has gone defunct, leaving behind a bunch of servers/storage arrays that would in normal circumstances get wiped/reprovisioned.
From the point of view of a customer of OSS (PBS), that's two extremely different things.
If it's scenario A, I don't see how PBS has any claim against Iron Mountain. Your typical datacenter colo host for bare metal hardware owned by a customer has no involvement whatsoever in the condition or operation of the data, operating system, filesystems, RAID arrays, ZFS, etc of how the customer has set up their environment. Nor any ability to do anything with it. A colocation host that hasn't been paid for its rack space and power will typically have clauses in its colo contracts allowing for seizure and sale of abandoned hardware after a certain period of time.
ApolloFortyNine 1 days ago [-]
>storing over 50TB of data
I thought this would be like the backups I've heard broadcasters have of every broadcast they've ever had (I've heard of people being able to request 20 year old news clips before).
But just 50TB of data? Duplicating that would have been cheap and trivial.
thekid314 1 days ago [-]
Yeah, we are talking about 3 20TB drives in a closet here.
I'm an independent photographer and I have my 20TB drives behind my monitor, backing up to backblaze with an offsite synology. Keeping this archive up does take time and money, but PBS should have the budget and IT deparment for this.
berkeleyjunk 23 hours ago [-]
I think for most of broadcasting history video was fairly low quality (according to today’s standards). So I am not totally surprised that this number is low.
slfnflctd 9 hours ago [-]
Yep. 1998 was when the first HD broadcasts started in the US, and it took a few years before everyone switched over. Before then, everything was 720 × 480 at 30 fps, and before the mid-80s (and for many shows for a long while after) the sound was mono, not even stereo.
Data requirements for this would be far, far lower than what is considered standard now.
WalterGR 23 hours ago [-]
> broadcasters have of every broadcast they've ever had
It depends - especially if you go back further than 20 years. The Wikipedia article for "Lost television broadcast" goes into detail:
> But just 50TB of data? Duplicating that would have been cheap and trivial.
Seriously. I'm just a random nerd and I have a home server with 96TB of raw disk space divided in to two RAIDZ1 pools which cost me around $2500 to build five years ago and has required basically zero maintenance since then.
Hard drive prices are absolutely bonkers right now but even with that in mind if I were to build a modern version of this same machine I could do it for under $10k using brand new hardware and it'd have double the storage
dylan604 22 hours ago [-]
Most broadcasters would consider their ultimate back up as the tape library/vault of their broadcast video tapes. Older stations will have a large collection of analog tapes. Most will only be digitized if someone requests it as capturing their entire library just to have digital files few people will ever want is an expense a station just isn't going to do for the lulz.
I wonder how they got the contract? Seems to be 4 people 'employed' at this company. Of which, one is retired, one is "Linkedin Member", one is a controller and the last one is a social media person: https://www.linkedin.com/in/isabelle-anderson-7aa250b5/
So, just the person to deposit checks and a social media person to get paid. Who exactly is left to work on storage infrastructure?
From the article it seems like Iron Mountain was doing the actual storage, and OSS was just acting as a middleman and taking some money off the top.
bityard 1 days ago [-]
To be somewhat fair, Iron Mountain is just a storage facility. OSS were likely the consultants who worked out how to get the archival data from the station's tapes into Iron Mountain and then get the data back out when needed. So they probably offered a worthwhile service, they just eventually stopped doing it and left the station in a lurch.
chuckadams 1 days ago [-]
And they sent OSS a legal demand letter to keep paying the Iron Mountain bill for their data. If OSS actually was using IM and not just running a scam, then the data should be pretty safe, if just not accessible for a few weeks/months.
pavel_lishin 1 days ago [-]
I guess that depends on how long Iron Mountain has to keep their data once OSS stops paying - and we don't know how long ago OSS's money ran out.
cwnyth 22 hours ago [-]
Also, my bet is that Iron Mountain significantly raised their prices and made it more difficult to retrieve the data without paying.
With the stuff about "corruption" and losing the data if the server is shut down, I wonder if part of the concern is about losing in-memory decryption keys.
bpodgursky 16 hours ago [-]
> William Cravens, the attorney representing Iron Mountain, told the judge his client doesn’t know the format of Nine PBS’ materials that were stored by OSS. He expressed concern about whether Nine PBS’ archival material is lumped together with data from other OSS clients. Iron Mountain wants to avoid potentially corrupting the other data, Cravens added.
That's an entirely reasonable concern and explains why Iron Mountain wants OSS on the hook for fulfilling this order rather than trying to yolo it themselves.
arjie 1 days ago [-]
It's funny to me that entire companies exist that have a fraction of storage that I personally have in a datacenter on flash let alone in total. 50 TiB is so little. It's like $7k annually on R2 and under a thousand bucks on S3 Glacier Deep Archive. I suppose it's just whoever knew the cousin of whoever at whatever position of authority but it's funny how just using deep glacier means he would have such insane margins over paying Iron Mountain.
Melatonic 1 days ago [-]
Your assuming it's all digital and can be stored on drives. It might be a large amount of physical film reel, magnetic tape, and tons of mixed legacy formats
gpm 21 hours ago [-]
It's reported in bytes - thus it's all digital. It's a "cloud storage vendor" - thus it's all digital.
Nine PBS might well have non-digital data, but it's clearly not the data this article is about.
gpm 21 hours ago [-]
50TiB is like $4k consumer SSDs in a shoebox at this point (not actually the recommended solution, but...). And that's massively inflated thanks to the AI boom.
pavel_lishin 18 hours ago [-]
> not actually the recommended solution, but...
That $4k plus the free shoebox could have saved them quite a bit in lawyer fees :P
Tangurena2 8 hours ago [-]
> 50 TiB is so little.
Still, it is 70 years of video from that PBS station.
1 days ago [-]
amiga386 22 hours ago [-]
They should've kept the archives in their intern's car, like the State of Ohio did
Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
— Andrew S. Tanenbaum
Cram a few 12 TB drives into that sucker and you’ve got all the storage you need for their whole 50 TB archive.
annzabelle 21 hours ago [-]
I'm not sure about currently, but up until very recently, the IRS did offsite backups by sending trucks full of tapes down the interstate.
fragmede 20 hours ago [-]
...how else would you do it? The Internet is only so fast and if you don't care about latency that's what you'd do.
annzabelle 20 hours ago [-]
I guess I found the storage medium dated. Though, my understanding is that certain kinds of tapes are much better for true archival storage than HDDs or SSDs.
toast0 17 hours ago [-]
Plenty of modern tapes. LTO-10 came out last year with 30TB tapes and 40TB tapes came out Jan 2026. IBM has a propetary tape system with 50TB in a slightly larger format, that came out in 2023.
The drives are spendy, but $/TB on media is lower than anything else. 30-50TB tapes are still pretty expensive, but so is all storage these days (I hadn't really looked in a while, yikes)
unsnap_biceps 17 hours ago [-]
Tape prices are lower, but the drive cost is very outsized. The cheapest drive on B&H is $13,584.05
Dylan16807 12 hours ago [-]
The cheapest LTO-10 drive. They have LTO-8 drives for less than half that price. And LTO-8 tapes are significantly cheaper per TB because they're not brand new.
For some reason they also have an LTO-5 drive for $1200 and tapes for $20/TB. Which looks a lot better today in a comparison with hard drives than it did a year ago.
toast0 10 hours ago [-]
Yeah, I said they're spendy... But if you're filling trucks with tapes, the cost of the drive(s) will amortize pretty well.
Don't trust Internet Archive as your only backup method though. There has been instances where they delete files without warning with the only explanation being that they don't meet TOS
embedding-shape 1 days ago [-]
> Don't trust Internet Archive as your only backup method though.
Agreed, no doubt about it.
> There has been instances where they delete files without warning with the only explanation being that they don't meet TOS
What sort of content? All the content I've judged to be of public interest and uploaded to Internet Archive, is still online there today. But I'd understand that depending on the type of content, that might not be true for everythibng.
ChrisRR 12 hours ago [-]
From what I remember it wasn't anything out of the ordinary, like TV shows or software or something.
As far as I remember the issue may have been that archive.org already had a copy of what had been uploaded, but they deleted it without warning and then refused to offer an explanation
So it may have been a legitimate deletion, but poorly executed
fragmede 20 hours ago [-]
it's cop material and the owner asks them via DMCA to take it down
toomuchtodo 1 days ago [-]
Inability to access items does not mean deleted. In the case of PBS content, it is unlikely access would be lost. I cannot speak to your use case, but the Patron Help Desk can confirm if an item is relevant to their preservation efforts if you'd like to check before uploading. Email info at archive dot org to inquire.
Someone dumping their Twitch stream or random Youtube content into the storage system will certainly have that content deleted and be banned if prior arrangements have not been made, for example. Don’t do that.
It's incredibly frustrating. The Internet Archive offers a Vault product, where you pay $2-3/GB one time for perpetual storage into the future of your collections, items, media, etc. This pricing was previously predicated on the cost decline curve of storage. It's gotten a bit more expensive due to the AI bubble inflating the cost of storage, but to your point, the Internet Archive is a superior custodian than some rando tier 2 cloud provider who won a contract from a non profit and will lose all of the data when they go bust.
Loses "access to" the data. The data is still there, they're just waiting for a court order.
craigmcnamara 21 hours ago [-]
Why should they have to get a court order when the party in custody said they have the data? Unless they don't actually have the data and they're hiding behind the court.
smallerize 20 hours ago [-]
Iron Mountain doesn't have it in the PBS station's account, it's in OSS's account. Instead of taking on the liability of deciding whether PBS should really have the data or if they're mistaken or even fraudulent, Iron Mountain is waiting for the court to sort that out.
Tangurena2 8 hours ago [-]
Their contract said that they had 30 days to get the data when OSS went bust. Instead, IM cut them off immediately. They need the court order to make IM comply with existing contracts.
UqWBcuFx6NV4r 21 hours ago [-]
Holy reading comprehension and assumption creep!
Nobody said that it was OK. But losing data and losing access to the data is clearly not the same thing. That’s why this clickbait headline decided to say the former instead of the latter, because it sounds worse. Read the article.
pudgywalsh 20 hours ago [-]
Because it's not their customer, the defunct company is.
If you had an account with them, how would you feel if some unknown third party could call them up and go "Uh hello you have my data give me access plz trust me bro"
ktm5j 1 days ago [-]
> This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.
Okay hang on now.. that strategy comes with its own risks. Risks that I imagine PBS was trying to avoid by using a third party that specializes in archival data storage. This is a silly takeaway.
Edit: I was too quick to comment. I think they are suggesting that the NAS should have been used in addition to the third party storage, which is actually a great takeaway.
jerf 1 days ago [-]
I read it as a statement of just how little data this is by modern standards more than anything else. This company is going out of business and PBS is losing this valuable data, and there's legal drama and multiple corporate entities involved, and if you read all that you might think this is some amazing amount of data that is actually difficult to deal with. Instead it's something I could stroll down to Best Buy right now, plonk down my personal credit card, and walk out with enough stuff to do a credible backup job myself, and we're not even challenging my credit limit or anything.
10 years ago, this was a lot of data. It isn't anymore and it makes it that much more bizarre that they're having that much trouble getting it back. It would be less to just give them access and let them download it themselves and perhaps stick it somewhere else temporarily than to pay lawyers to deal with legal filings.
To put it in vaguely physical terms, what once was a university library is now a good solid bookshelf full of stuff. Legally wrangling over a huge library is one thing, but if it's just a bookshelf, metaphorically, let them come in for an hour and clean it out themselves rather than bringing in the law. Obviously not a perfect metaphor on data size, this is more about how lawsuits are more of a hassle then just giving it to them.
ameliaquining 1 days ago [-]
The reason nobody's just giving it to them isn't because the amount of data's too much of a hassle, it's because nobody can figure out who has the keys to the now-defunct storage company, and Iron Mountain quite reasonably won't hand out their customer's data to a third party without a court order.
jerf 1 days ago [-]
Yes, I agree completely. It just heightens the pathos of the story that it is over what is now such a small amount of data.
Tangurena2 8 hours ago [-]
> 10 years ago, this was a lot of data.
And this data is 70 years of video.
mindslight 1 days ago [-]
> 10 years ago, this was a lot of data.
More like 20 years ago (sorry). 10 years ago this would have been slightly stretching, but not by much - 11x8TB raidz3. Shuckable drives would have cost $3300 (plus the chassis), and already out of the husk probably 1.5x more. So let's say $5500 total even back then. (I built my current array around the same time)
Funny, these days you could do it with 4 spindles (4x24TB raidz2) but with the rampant price inflation in storage, you're still looking at $4500.
js2 1 days ago [-]
I recently pulled up my resume from 25 years ago and came across this amusing (by today's standards) tidbit: "Re-located a web-server farm between two data centers with zero downtime. This included migrating over 150GB of data between three Network Appliance Filers and moving more than 12 racks of equipment."
A whopping 150GB!
ndiddy 1 days ago [-]
> It's not clear if the channel has backups of its data, but based on its determination, we suspect not. This is strange, though, especially for a major organization, as one can easily purchase a six-bay NAS on [Amazon] and populate it with [the best HDDs].
I think they just put that stuff in so they could shill their affiliate marketing articles rather than out of any actual concern.
MostlyStable 1 days ago [-]
Having that be there only backup solution would have been silly and dangerous. Having it be part of their overall backup solution including with something like the provider would have been smarter and better and prevented this entire problem
mcmcmc 1 days ago [-]
3 copies, 2 different media, 1 offsite. One backup is none
anonymars 1 days ago [-]
Importantly, "one cloud vendor" is also none
embedding-shape 1 days ago [-]
1 copy, one other media, is better than 0 copies. Clearly 1 != 0.
bblb 16 hours ago [-]
If you had zero copies, then you'd have no data.
The 3-2-1 includes the production data itself.
You have one copy of data (the data that needs to be backed up), and you have two additional copies of that data (the "3" in the strategy), where one of the backup copies is stored offsite (the "1" part), which naturally follows that it satisfies the middle "2" also.
For over a decade we used 4-3-2 strategy, but it got expensive fast and due to ever evolving budget cuts, our (local gov) customers weren't ready to pay the extra.
There's also 3-2-1-1-0, but that's just a fancy way of saying you _really_ should be doing those restore tests you never seem to have the time, because ransomware can and _will_ corrupt your backup copies.
edit: the absolute minimal setup that still satisfies the 3-2-1 rule: 1) you have your data in production directory on local NVMe stick inside your computer, 2) you copy the original production data to another directory inside the very same NVMe, 3) you occasionally manually copy the production directory data to USB and store that USB inside the shed in your backyard. - That's it. Now your home data is covered by the coveted "3-2-1" enterprise ready backup strategy.
remuskaos 1 days ago [-]
I first read that as 0! = 1, which, incidentally, is true.
thebitguru 1 days ago [-]
The point still stands, but in what world can you build a 50TB NAS for $1,000 these days? I want to go there
nairboon 1 days ago [-]
$1000 is for the NAS, they still need to populate it with a couple of drives.
dghughes 1 days ago [-]
Yeah I bought a five bay Synology NAS two or three years ago I think it and three 18TB WD Red drives total cost $3,000CAD? I was going to save up for two more drives. Now each drive is $1,000 CAD a 150% jump. edit: I checked $1,350 each so about 250% jump ugh!
echoangle 1 days ago [-]
Technically you're still correct even when you would advocate for both. Every additional copy increases the risk that the data is accessed by someone else.
Tangurena2 8 hours ago [-]
It was 70 years of video that they had broadcast to the public. This isn't anything remotely proprietary.
1 days ago [-]
mulmen 1 days ago [-]
What risk? The point of PBS is “other people access the data.”
OutOfHere 1 days ago [-]
A NAS is at risk from malware that can erase all of it. 100GB Bluray discs exist and work more easily for archival storage, as does multi-terabyte tape drive storage, both of which can stay disconnected At least two offline copies, combined with error correction codes, are advised. A cloud backup is also advised.
Melatonic 1 days ago [-]
It's really too bad the disc based storage options made after Blu-ray were discontinued. They made a lot of progress and seem a lot more user friendly than tape.
tim-- 15 hours ago [-]
Sony's Optical Disc Archive (https://pro.sony/en_AU/products/optical-disc) is alive and kicking. Each cartridge basically uses 11 optical discs for 5.5TB per Cartridge and ~300MB/sec transfer speed[0]. It's last update was in 2019. Too bad we won't see a new generation, as Sony has basically killed any future development.
Folio Photonics has just got some funding and they are saying they will be able to launch with capacities of 10 TB per cartridge.
The situation is something we all will have to deal with as time goes on - this is an archive. If I remember rightly Facebook used to archive by just putting the old hard disks in a room. At the volume of data we are heading towards most successful companies will be in the hard position of finding some way of keeping data for the lifetime of a human being - this is hard
_carbyau_ 19 hours ago [-]
I get why having an offsite backup is handy. But for less than $10k you can easily get a 4 bay NAS with 96TB storage for a local backup.
This smacks of saving pennies on IT to spend dollars on lawyers.
ninalanyon 21 hours ago [-]
Such data should never be held by a third party unless it is one of several duplicates.
I keep photos on Google and MS but they are short term storage only and never the only copy.
nashashmi 21 hours ago [-]
Yeah well you are not holding 90 TB of data based on the decisions of a chain of supervisors.
People like me who no longer pay attention enough after seeing so much upheaval in the cloud space are losing out on ideas of best practice and giving in to whatever works for the time being.
Razengan 21 hours ago [-]
I've kept all photos on iCloud since decades; there's too many to move elsewhere and keep updated, but the local copy on my Mac is regularly backed up to a Time Machine disk.
adolph 1 days ago [-]
Not PBS but an affiliate broadcaster and the data is not lost as in destroyed, but legally encumbered.
Full headline: PBS broadcaster loses access to 50TB of data comprising 70 years of TV history after contracted cloud storage vendor goes defunct — public TV channel sues Iron Mountain data center, which hosts archival materials, to ensure preservation
The channel’s attorney also reached out to Iron Mountain to inform the company of the pending case in Colorado; Iron Mountain finally admitted to holding the data. Iron Mountain initially agreed to turn over Nine PBS’ data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.
ApolloFortyNine 1 days ago [-]
> Iron Mountain initially agreed to turn over Nine PBS’ data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.
This does make sense. I'm guessing the first request some rep just went sure its you're data why not, but in reality, it's OSS's data, and you paid OSS to store it. It's also likely encrypted in someway.
TLDR still, is especially when it's a reasonable amount of data, just have more than 1 backup.
RevEng 1 days ago [-]
I store my belongings with a storage company. They rent facilities from a property owner. The storage company goes under while my stuff is still in the storage unit. I go to the property owner and say my stuff was in there. They say, no, that's owned by the storage company. Incorrect - the belongings were owned by me the entire time, the storage company was just holding it for me. I need to prove it was mine, but I don't lose property rights just because it's in someone else's possession.
Likewise if I store data with a cloud provider. Sure, Iron Mountain shouldn't just offer it up without proof, but if I give them notice of a legal case they should be sure to hold on to it, and if I win the case they need to hand it over to me. All I need to do is provide proof that I am the rightful owner of those goods.
Tangurena2 8 hours ago [-]
> I store my belongings with a storage company. They rent facilities from a property owner. The storage company goes under while my stuff is still in the storage unit. I go to the property owner and say my stuff was in there. They say, no, that's owned by the storage company. Incorrect - the belongings were owned by me the entire time, the storage company was just holding it for me. I need to prove it was mine, but I don't lose property rights just because it's in someone else's possession.
You are a subtenant. The tenant went bust. The case law on this is settled. Depending on state, you either have a period of time to vacate the premises or you can/do become a tenant of the property owner. People have been subleasing rooms/apartments for hundreds of years, so every possible scenario you could imagine in this space has been litigated to death.
That is real estate.
American case law about data stored on computers is that whosoever owns that computer owns all the data on that computer. Which is why US privacy law is far weaker than EU privacy law (where you own the data about yourself). I think pets.com was one of the very few exceptions where the TOS said "we won't sell your data" and the bankruptcy liquidator said "it is an asset, we sure will sell it" and the courts said "no you can't sell it".
ApolloFortyNine 1 days ago [-]
In the edge case that it wasn't Nine PBS requesting the Nine PBS data, you've now opened your company to a huge can of worms. Only OSS knows who Nine PBS is in their database, hence why, from a liability standpoint, it makes sense to say it's OSS's problem.
Also your storage company case is actually it's own can of worms, see the Lego drama from a couple months ago about that. It seems obvious, but in the eyes of the law there's more paperwork than you'd think.
12_throw_away 19 hours ago [-]
You know how they say "possession in 9/10s of the law"? I.e., "9/10s of the law is just devoted to figuring out who owns what."
That is to say - I really doubt anything about this is remotely simple or straightforward, but I'm sure it's all lucrative for the lawyers involved.
gwbas1c 1 days ago [-]
When I worked for Syncplicity, we offered an on-site version for "cloud" storage specifically for this reason. (Ok, not specifically for this reason, it was because customers demanded it and an on-site version of our product was a great differentiator in our niche.)
roughly 1 days ago [-]
Calling out Nine PBS for not accounting for the fact that their vendor could effectively just decide to commit crimes to get out of their contract and their vendor’s vendor could just decide to compound the problem in the face of a court order is wild. Yes, we’ve all learned to anticipate everyone else on the planet being an absolute shitheel; no, Nine PBS did not do anything wrong by assuming the professionals they were paying to render a professional service to them would do so. We’re all paranoid engineers, but this is absolutely no goddamn way to run a society.
manarth 6 hours ago [-]
> "their vendor could effectively just decide to commit crimes"
Going bankrupt isn't inherently criminal. And IM are likely to be within their contractual rights to lock the OSS account if IM isn't being paid.
> "their vendor’s vendor could just decide to compound the problem in the face of a court order is wild"
The initial court order was between "Nine PBS" and "OSS". Iron Mountain were not a party to the initial court order, and granting access without a court order naming Iron Mountain could open them up to liability from OSS.
This looks like a similar formality of following the process so that a court legitimises the transfer of data back to Nine PBS.
roughly 1 hours ago [-]
I'd suspect somewhere in the contract between Nine PBS and OSS there's some wording about what happens in the event of OSS discontinuing operations that doesn't include the words "silently fuck off into the woods without informing customers," which appears to be what happened here.
I agree that IM is probably inside of their contractual rights, but they're still handling this in a fashion that risks causing severe issues for Nine PBS. The presence of the court order seems like sufficient inducement to at least hold the deletion until the court decides, and I'd be very curious what kinds of legal remedies Nine PBS might be entitled to should IM not do so and the court decide in their favor.
1 days ago [-]
Good4boothee 1 days ago [-]
Isn't 50TB just 6 years of videos, in even the best case (1GB/h)?
undersuit 1 days ago [-]
This local PBS wasn't playing it's own footage 24/7. The data in question could be just their local segments, an hour or two of local coverage and maybe their fundraisers.
reaperducer 24 hours ago [-]
Isn't 50TB just 6 years of videos, in even the best case (1GB/h)?
They weren't recording in 4K last century.
One hour of 1970's-grade NTSC video is around 300MB.
More to the point, it's not like they archived 70 continuous years of video. Many local PBS stations produce an hour or less of video each week. This being a St. Louis station, however, it's probably somewhat more.
kristopolous 21 hours ago [-]
The magic of public-private partnerships strikes again.
Like most of the time, this will probably be held for some kind of extortionist legal ransom. We just love handing our stuff over to bad-faith third-parties in the name of some kind of imagined freedom. Just can't get enough of it. Public infrastructure, national archives, city services, we just seem to be completely deer-in-the-headlights when they pull this stuff on the daily...
So here we go again. We blindly trust in private parties always magically acting in good faith when there's billions on the table incentivizing them not to.
We've been doing it like this for 40 years and it hasn't worked, but I'm sure our ideological commitments will bend reality eventually.
This is your money btw.
dgoldstein0 20 hours ago [-]
Pbs is offering to pay for the recovery, but the company who supposedly has the data basically went non responsive. Honestly this sounds like a mishandled going out of business where the company left with the assets was not prepared to deal with the customers at all.
kristopolous 20 hours ago [-]
We'll see. It's crazy that we still have lots of institutions that would have taken this seriously.
County libraries, city hall records, large public universities.
It should have been like "hey, UNC Chapel Hill" or "Los Angeles County" take care of this.
I mean maybe things like that happen, like UTexas just loses multiple shelves mysteriously in their special collections, but I've never heard of it.
I just hate when ideological commitments shove reasonable solutions like this off the table
ggm 23 hours ago [-]
Intermediaries. When it's not the DNS it's the anomalous status of intermediaries.
dehrmann 1 days ago [-]
> since 2019
At that point, why wouldn't you pick a major cloud vendor? This feels like more of a governance failure.
mulmen 22 hours ago [-]
Because of the next sentence in that paragraph:
> This unidentified vendor provided “hardware, software and cloud-storage services” for storing the public broadcaster’s archival materials and other data.
orionblastar 21 hours ago [-]
I don't trust cloud services; I bought a 12T USB hard drive for my backups.
brianwawok 21 hours ago [-]
Statistically a fire is more likely to burn your 1 copy than the data center I bet. Ideally you have both, that survives almost anything outside thermonuclear war
dboreham 21 hours ago [-]
Probably should buy three, and put them in places far enough apart that an earthquake or large fire won't get all of them at once.
paxys 22 hours ago [-]
All this over 50 TB of data. Could have created an in-house replicated NAS with like $2500 worth of hardware.
jdboyd 22 hours ago [-]
By my calculation using what I can find of 2019 pricing, a more reasonable solution would have been:
Synology RackStation RS1219+ - $1,200
8x Ironwolf Pro 12TB drives - 8x $450 = $3600
so for a usable RAID-6 capacity of 65TB about $4800 + OSS, or double that if they have two locations they could and replicate between them.
I know there would be cheaper hardware options available (I'm partial to older surplus enterprise hardware myself) but it isn't particularly reasonable to expect a small business to choose a more DIY (at least in terms of software configuration) option. You could step down to 10TB drives, but it seems unlikely they would have wanted to risk that not being quite enough.
On top of that, since 2019 is 7 years ago, it is reasonable to expect they would have had a hardware refresh at some point, I would think 5 years would be more reasonable than 3 year for something like this. At that point, the nas would be more ($1300) but the drives would be less (8 * $300), so add either another $3700 (double if both are replaced, but maybe they figured the offsite backup could be stretched even longer than 5 years)
So, rather than $2500, it would probably have been more like $8500 + OSS or $13,300 without OSS. Still that would have meant a proper are archive and a proper backup and they wouldn't be in this mess, so I completely agree with you about what they should have done, but think you were underselling the price a good bit.
I work in several labs where I have to give this advice, which some labs are smart enough to make an ersatz backup that we store somewhere (PI's house) as a cheap offsite. Even a partial recovery is great in the face of fire/flood/explosion, or the most common error: IT fucks it up and then says "oh well".
Hope someone has some copies and they can rebuild, what a nightmare.
RIMR 1 days ago [-]
From the article:
>This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.
Look, I know that a certain percentage of Tom's Hardware articles are just sales pitches for a NAS, but this is asinine. Iron Mountain deals largely in tape storage. 50TB of tapes fits in one of their small archive boxes. If they're using the latest tech (LTO-9), it fits on 3 tapes.
This is exactly how you are supposed to store archival media for decades, not on a NAS. A NAS is for storing the files you need to actively access.
Melatonic 1 days ago [-]
How do we know the content isn't on BetacamXR or straight film reels ?
eviks 1 days ago [-]
Not really, you're never "supposed to" rely on a single company with 3 tapes for your only copy
blltprfmnk 1 days ago [-]
Title should be updated to reflect the fact that this is a single PBS station.
cootsnuck 1 days ago [-]
That made me feel relief until I realized it's my own local PBS station.
toomuchtodo 1 days ago [-]
Any interest in connecting me with someone there so I can attempt to obtain a copy for the Internet Archive?
ButlerianJihad 1 days ago [-]
Ah, "Channel Nine PBS", not a cluster of nine pbses
hnburnsy 1 days ago [-]
Pretty clear that this is AI copied slop that no one at Toms Hardware bothered editing as it has no clarity that Nine PBS is the PBS affiliate in St Louis, Missouri.
(I've submitted this to HN's mods via email. The original submission was to a very poor TomsHardware link.)
OT: The tendency of local broadcasters to fail to identify either on websites OR QUITE OFTEN ON BROADCASTS THEMSELVES even vaguely where they operate is ... a long-standing annoyance of mine. Sure, local audiences may know, but those passing through, or catching the signal at a distance, likely won't.
I've spent more than a few nights at higher elevations (campgrounds, highways) listening to clear channel stations reporting on weather and traffic "in the local area", sometimes for hours, without ever hearing what "the local area" is. That for signals travelling hundreds of km, and not infrequently multiple thousands of km.
(Don't get me started on entire countries where stations are identified simply by number, e.g., "Channel 4" in the UK.)
tomhow 23 hours ago [-]
Link updated, thanks!
puddnutz 23 hours ago [-]
It's actually funny you mentioned the lack of indicating call sign or operating location. I saw something similar done on purpose.
I worked at WCCO in Minneapolis briefly a few years ago. Right around that time, CBS corporate was attempting to standardize all their stations and associated websites. In their mind, that meant making everything "CBS News [Location]" and stripping just about everything else from the public marketing.
WCCO, KPIX in Los Angeles, and a few other stations battled back about this because the local name recognition of the call sign was stronger than CBS would be by itself.
So now, when you bop around the "Local News" section of the CBS Website, some cities have logos with the call letters, and some do not.
In short, when you see a station that that's difficult to identify, try finding out who owns it. Very often the ownership is to blame for one reason or another.
The United States' Federal Communications Commission (FCC) enforces specific requirements for identification that must be followed by all terrestrial radio and television stations. Stations must, when they sign on, sign off, and as close to the top of each hour as feasibly possible (such as within a "natural break" in programming, like a commercial break), present a visual (television) or aural (radio) station identification that contains, at minimum, the station's callsign, followed by its designated city of license. As a courtesy, top-of-hour identifications may also contain additional information, such as frequencies and a declaration of the station's ownership.[6] Only the name of the licensee, the station's frequency or channel number as stated on its license, and/or network affiliations, may be inserted between the call letters and station location.
Technical information is provided. This can include station identification (call sign and city of license), transmitter power, frequency or channel number, translators used, transmitter locations, list of broadcast engineers, and/or studio/transmitter links (STL).
In my experience, the 1980s were a time when some TV stations would still sign off for a few hours, perhaps only on Sunday night. It is unusual today in the United States, for commercial radio or TV stations to sign off and sign on. When stations did sign off/on, I did hear some of them broadcasting an extended Station ID. A man's voice would call out those details listed. They would mention what mountain peak their main transmitter was on, and the wattage of "peak effective radiated power". Since I lived near the Mexican border, there were stations with Mexican licenses who broadcast in English programming, except for the Mexican National Hour. "Border Blasters", as they were called.
dredmorbius 21 hours ago [-]
I'm aware of station identification requirements. It's possible that there were top-of-the-hour announcements I didn't hear or don't remember.
What I distinctly recall was that it was possible to get through a full weather or traffic report without the region being identified.
These days, a frequent gripe is that local-media websites (radio and television) fail to specify the state. Where a call-sign is available, I'm obliged to look that up on Wikipedia to figure out what location the story concerns. Place names are often given only as small town or county names, neither of which typically have national recognition, and many of which repeat from state to state.
Just email…
https://rsync.net/pricing.html
Which, in practical terms, means "while I or my investors have money". These guys have 70 years worth of data, can you guarantee to keep it for another 70? The real answer to these kinds of problems is managing the data yourself
Libraries were the archives for newspapers and local public information.
Taking its name (Lots of Copies Keeps Stuff Safe) from the idea that a file is more likely to survive if it has multiple copies stored in multiple places, LOCKSS software allows libraries to create preserved digital collections out of materials that would otherwise be accessible only through a licensed academic subscription. LOCKSS has agreements with academic publishers to permit libraries to store local copies of content that is usually only accessible through web-based subscriptions, enabling post-cancellation content access.
https://coptr.digipres.org/index.php/LOCKSS_(Lots_of_Copies_...
Edit per “no contact on profile” comment:
Storytime, since I'm no longer under NDAs for the company, over a decade ago somewhere that I worked, a disgruntled employee who decided to quit, made a point to within a week of quitting, take a whole bunch of important physical documents they had (because, FBOW, our email and other data retention was so bad, it was safer to have them as prints) and threw them all in our fancy Iron Mountain 'disposal' bin.
Apparently due to the agreement with them, short of a court order, only specific employees/titles in the company were allowed to be the persons to retrieve them.
Keeping that situation in mind, it's clear that this dance for better or worse is part of their posturing as a retention center with very strict rules to ensure compliance while also avoiding snooping.
Anything else is the sort of stuff that causes blog posts and news articles about data breaches to be written.
I've dealt with them as well - they don't like being in this kind of position either.
[0] - https://ironmountain.jobs/manassas-va/armed-officer-safety-s...
[1] - https://ironmountain.jobs/boyers-pa/security-officer-armed/6...
They weren't talking about breaking into the Iron Mountain facility
It’s exactly like the above but it’s presented as if the data sits neatly on two drives stored in a safe and all you have to do is mount them to see the unencrypted data.
Consider this: the utility that provides the electricity and the ISP providing internet also do not have a direct relationship with Nine PBS, but will be essential in carrying out the court order. Does the judge have to expressly name every transitive party responsible for carrying out the order? Time will tell, but to me the answer seems "no."
I'm sure Iron Mountain will be happy to turn over the data, but to cover their ass in terms of their agreement with OSS, they need the cover of a court order otherwise they would likely be in breach of their contract with OSS
Iron Mountain just doesn’t want to be in the middle, and I can see why. OSS went defunct, PBS files a lawsuit, guy buys OSS so PBS pauses, guy now claims he was defrauded into buying it, and somewhere in the middle nobody from OSS showed up in court so there’s a default judgement to turn those over.
I would probably also refuse to turn over the documents without a court order. The whole situation is a mess and the only way to be absolutely sure you aren’t going to get dragged into it is a court order for specific performance.
PBS probably isn’t going to sue them for releasing the docs, but who knows what OSS will do. It doesn’t sound like anyone even really knows who owns OSS at the moment.
In fact, I'd expect you to need a seizure-warrant/distinctly-named-but-to-the-layperson-identical-court-order for my place as, among other things, I don't think I am in a position to legally have standing to challenge an order that's not mentioning me, and lack of me being an active flight risk would probably preclude you from arguing exigent circumstances to secure the dev kit without a judge ordering me to let you.
TL;DR: "complying" with a court order that's filed against a 3rd party is, IMO/AFAIK, a voluntary act, specifically because the party not mentioned in the court order AFAIK has no legal standing to challenge that very order of in court. _Precisely_ because they're not legally mandated to comply, at least not beyond perhaps taking a precautionary legal hold on the referenced information to not get accused of aiding in destruction of data in case the court-ordered party directs the non-mentioned party to delete through the regular customer relationship channels (API? Whatever else?).
That's making a lot of assumptions. I host some of my backups on what is effectively a reseller of S3; was that decision corrupt?
But, yes, PBS does bear some responsibility here. If you leave your car in a parking lot overnight with the door open and the key in the ignition, the thief is still wrong for taking it, but there's a limit to how much indignity you can have.
PBS is an association of local public broadcasters, most of whom are SEVERELY underfunded.
As the parent entity, the national PBS has a responsibility to guide and even mandate policies and provide best practice guidelines or even offer support and assistance in burden sharing arrangements.
PBS has long now had major structural challenges due to things like internal corruption and overbearing control by the well funded chapters without commensurate responsibility.
As with many such decentralized membership organizational structures, incentives can often be very misaligned between verticals and laterals.
Don't blame corruption for what was most likely extremely strained budgets. Local public access TV stations like KETC (Nine PBS) tend to be severely understaffed and underpaid.
Realistically, they went with OSS because the quote for a local MSP was cheaper than the cost of hiring additional headcount (a $70k salary ends up costing roughly $100k for an employer).
(Source: occasionally dealt with these kinds of accounts back when I was an SE and later PM)
Edit: I'm right - they only have 1 IT/Software employee managing the entire organization which only has a budget of $30M after losing $1.8M in federal funding.
Oof, that is a horrible position to be in with so many dicks on HN. You'd assume SWEs would have sympathy for other SWEs.
Some HNers should have been taught better manners by their parents or should have at least internalized the lessons from Arthur as a kid [0].
If that employee sees this post - I want to say THANK YOU. It's a hard job.
[0] - https://arthurwiki.com/wiki/Go_to_Your_Room,_D.W.
Only if you haven't been reading HN very long!
I hate HN and how it has gotten worse and worse as the years go by. There are a couple basic changes that the mod team could do to help alleviate the degradation but they aren't doing it.
You're going to have to unpack the assumption here: are you alleging that there was a kickback in the procurement process?
If you don't want on site, a second off site provider, 50TB on backblaze would be an extra ~$350 per month (prob less if they were to commit to capacity instead of using their PAYG service)
I wasn't 100% sure, but the data backs this up[0]; the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.
- [0] Interestingly, Ngram (https://books.google.com/ngrams/graph?content=%5B3-2-1+backu...) says ~2010 while Trends (https://trends.google.com/explore?q=3-2-1%2520backup%2520rul...) says slow build up from ~2020, guess "everyday" language would sit somewhere around those two different truths.
Also, this TV station has not been sending their show archives to Iron Mountain for the past 70 years. Probably what happened was: In 2019, they had an on-site or nearby archive. Then some enterprising IT director came along and convinced management that they would save a bunch of money and time by outsourcing everything to a third party who said they could do it for cheap. Very confident in their genius, they then scrapped their own archive entirely.
Part of me pities them for ending up in this situation, the other part of me believes they got everything they asked for.
Which is not as thorough, but still makes the point that you should do more than one backup
Probably! The important part is that your pretend story time supports your base desire to blame the victim!
While statistically it might equate to "unlikely" as you state, the fact it did happen, I now have that experience that is not forgotten. Never bring a large storage pool into PROD immediately after the array finishes striping. Put it under load for a period of time first. The question remains on how long that period is though which is probably answered with "how paranoid are you"
While there's always luck involved, dying during warmup and not two weeks later is the expected experience for drives that defective.
So what happened to you wasn't that you made a backup and then it failed so fast you couldn't use it. It's relevant to what I was saying is unlikely, but it's not an example of it.
Plus at zero week or even two weeks you have the previous backup system around. If you have a big simultaneous failure 6 months out, that's a scary situation, but that's much rarer.
They signed up with them in 2019. Not long ago.
Even if it is a "new" concept, common sense isn't. Pretty much anyone who cares about their data will do the due diligence, and this is not something that requires technical chops. This has been a concern for well over 20 years!
It really sounds like some senior management trying to be cheap.
They probably have 1-2 IT staff, who have been paid significantly under market value compared to private sector.
Management was not management of an IT organization, and probably did not really understand the trade-offs or what they'd lose by not spending more money on IT. Their IT staff was probably super busy and over-worked and also (not to throw them under the bus) not experienced enough to really advise them, and if they had and had tried to advise them, would management have decided that doing less programmatic work to pay for it, and having IT people at least temporarily spend less time on whatever they do to support regular business, to set up and support backups, was worth it? Maybe, maybe not. Again, the managers probably have little experience with IT. The whole organization is probably well under 100 people.
Source: I work in non-profits.
Why is HN always focused on suggesting there's a person who could have easily made the right choice if they weren't an idiot?
I'm familiar with the dynamics of such places.
If they actually go through a proper lawsuit, my guess is the lawyers will cost more than whatever they'd have to pay for redundant backups by an order of magnitude. Which means they probably won't go to court.
I've heard variations of this all the way back to some 1990s vocational IT courses in the country I live, and it wasn't even technologically advanced at the time. It's hard to have any sympathy for someone who misses this concept in today's age.
I mean, that 70+ year old archive must have been digitised at some point, unless Iron Mountain was storing the reel to reel tapes and/or film from the original recordings. So some "modernation" has been applied to it since its creation.
Now add in tons of other legacy formats and digitising becomes a pretty big cost for a public station.
Sounds like they paid a company to digitise all their old tapes, and store the results. They possibly still have all the tapes (on site) and are just trying to avoid paying the costs to redigitise them.
But, it’s also quite likely they destroyed all original the tapes after digitisation, and this was the only copy.
https://www.ibm.com/history/magnetic-tape
Usually what kills the tape is the little reflective stickers on the back side of the tape material will fall off and render the tape useless since the machinery can't tell where the recording starts. Apparently I bought cheap mag tapes back then.
At 50TB, they probably could have Just™ written three copies of the data to a pile of disks, sent one copy off to Iron Mountain, and kept the other two copies unplugged in a safe. I can understand the DIY solution might not be allowed for insurance reasons, but it's something so cheap they probably could have done both. Unless OSS was just taking them for a ride on storage, which... well, actually that seems pretty likely.
[0] To put it simply, there is no good medium for the archival of digital information. HDDs are not shelf-stable. Flash is even less shelf-stable than disk. Sony is actively trying to kill optical. LTO is shelf-stable, but good luck buying known-good or new drives for old formats. The only option is periodic cloning and verification of offline disks, or keeping all your disks online, which is the NAS solution.
IM seem to be holding KETC's data hostage, which is Not Cool.
Maybe they mixed data from many companies on the same volume like S3 does. Maybe there were ACLs on that. Maybe it includes PII. Maybe it's a bunch of shards and needs an index on OSS servers to reassemble. Maybe it's encrypted by keys OSS had.
2. That doesn't actually seem to be Iron Mountain's argument here, though the reporting isn't crystal on the point.
There is no way for Iron Mountain to determine who OSS’s customers are, and their permissions, without help from OSS itself.
And because OSS is defunct, they can’t get that help.
How does AWS then determine which files are whose when my angry clients go knocking on their doors because I've decided to disappear?
Non-profit IT has always seemed, to me at least, to be a land of compromises. There's rarely enough money, knowledge, or time to do things "the right way", so you have to do the best you can. If I could only choose one backup to have, it'd be remote on a provider like Iron Mountain. It would seem as if they are storing things on Iron Mountain through an intermediary company, who may or may not have actually paid the bill.
So, y'know, fraud's a possibility here.
But also, from that link...
"Americans' views on gun control seem to stem almost entirely from their cultural worldview and how they position themselves in that cultural schema. Statistics do not persuade them to change their minds."
If true, stick a fork in us, we're cooked.
In a free society, we don't need state-run media.
I think this is a kind of time-sequence fallacy which I've seen before on HN. It's not that "we don't," but that, perhaps, "we wouldn't."
E.g., I study the pieces still standing in famous checkmate positions and decide to start the game using only those pieces. Bad strategy.
The US is not a "free society".
Which is why the BBC, ABC, CBC, etc exist in civilized societies.
a) OSS is a colocation customer with its own hardware colocating it inside an Iron Mountain datacenter. In which OSS owned the bare metal and paid iron mountain for rack space and power.
or
b) OSS is a dedicated hardware customer of Iron Mountain running a service on bare metal owned by Iron Mountain, and has gone defunct, leaving behind a bunch of servers/storage arrays that would in normal circumstances get wiped/reprovisioned.
From the point of view of a customer of OSS (PBS), that's two extremely different things.
If it's scenario A, I don't see how PBS has any claim against Iron Mountain. Your typical datacenter colo host for bare metal hardware owned by a customer has no involvement whatsoever in the condition or operation of the data, operating system, filesystems, RAID arrays, ZFS, etc of how the customer has set up their environment. Nor any ability to do anything with it. A colocation host that hasn't been paid for its rack space and power will typically have clauses in its colo contracts allowing for seizure and sale of abandoned hardware after a certain period of time.
I thought this would be like the backups I've heard broadcasters have of every broadcast they've ever had (I've heard of people being able to request 20 year old news clips before).
But just 50TB of data? Duplicating that would have been cheap and trivial.
I'm an independent photographer and I have my 20TB drives behind my monitor, backing up to backblaze with an offsite synology. Keeping this archive up does take time and money, but PBS should have the budget and IT deparment for this.
Data requirements for this would be far, far lower than what is considered standard now.
It depends - especially if you go back further than 20 years. The Wikipedia article for "Lost television broadcast" goes into detail:
https://en.wikipedia.org/wiki/Lost_television_broadcast
Seriously. I'm just a random nerd and I have a home server with 96TB of raw disk space divided in to two RAIDZ1 pools which cost me around $2500 to build five years ago and has required basically zero maintenance since then.
Hard drive prices are absolutely bonkers right now but even with that in mind if I were to build a modern version of this same machine I could do it for under $10k using brand new hardware and it'd have double the storage
I wonder how they got the contract? Seems to be 4 people 'employed' at this company. Of which, one is retired, one is "Linkedin Member", one is a controller and the last one is a social media person: https://www.linkedin.com/in/isabelle-anderson-7aa250b5/
So, just the person to deposit checks and a social media person to get paid. Who exactly is left to work on storage infrastructure?
https://www.linkedin.com/company/osstorage/people/
With the stuff about "corruption" and losing the data if the server is shut down, I wonder if part of the concern is about losing in-memory decryption keys.
That's an entirely reasonable concern and explains why Iron Mountain wants OSS on the hook for fulfilling this order rather than trying to yolo it themselves.
Nine PBS might well have non-digital data, but it's clearly not the data this article is about.
That $4k plus the free shoebox could have saved them quite a bit in lawyer fees :P
Still, it is 70 years of video from that PBS station.
https://www.newson6.com/story/5e3687772f69d76f6209d656/tape-...
Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
— Andrew S. Tanenbaum
Cram a few 12 TB drives into that sucker and you’ve got all the storage you need for their whole 50 TB archive.
The drives are spendy, but $/TB on media is lower than anything else. 30-50TB tapes are still pretty expensive, but so is all storage these days (I hadn't really looked in a while, yikes)
For some reason they also have an LTO-5 drive for $1200 and tapes for $20/TB. Which looks a lot better today in a comparison with hard drives than it did a year ago.
Agreed, no doubt about it.
> There has been instances where they delete files without warning with the only explanation being that they don't meet TOS
What sort of content? All the content I've judged to be of public interest and uploaded to Internet Archive, is still online there today. But I'd understand that depending on the type of content, that might not be true for everythibng.
As far as I remember the issue may have been that archive.org already had a copy of what had been uploaded, but they deleted it without warning and then refused to offer an explanation
So it may have been a legitimate deletion, but poorly executed
Someone dumping their Twitch stream or random Youtube content into the storage system will certainly have that content deleted and be banned if prior arrangements have not been made, for example. Don’t do that.
https://old.reddit.com/r/DataHoarder/comments/sq6wbq/please_...
https://archive-it.org/vault/
(huge fan of Stokes as a fellow archivist, recommend the documentary on her: https://en.wikipedia.org/wiki/Recorder:_The_Marion_Stokes_Pr...)
If you had an account with them, how would you feel if some unknown third party could call them up and go "Uh hello you have my data give me access plz trust me bro"
Okay hang on now.. that strategy comes with its own risks. Risks that I imagine PBS was trying to avoid by using a third party that specializes in archival data storage. This is a silly takeaway.
Edit: I was too quick to comment. I think they are suggesting that the NAS should have been used in addition to the third party storage, which is actually a great takeaway.
10 years ago, this was a lot of data. It isn't anymore and it makes it that much more bizarre that they're having that much trouble getting it back. It would be less to just give them access and let them download it themselves and perhaps stick it somewhere else temporarily than to pay lawyers to deal with legal filings.
To put it in vaguely physical terms, what once was a university library is now a good solid bookshelf full of stuff. Legally wrangling over a huge library is one thing, but if it's just a bookshelf, metaphorically, let them come in for an hour and clean it out themselves rather than bringing in the law. Obviously not a perfect metaphor on data size, this is more about how lawsuits are more of a hassle then just giving it to them.
And this data is 70 years of video.
More like 20 years ago (sorry). 10 years ago this would have been slightly stretching, but not by much - 11x8TB raidz3. Shuckable drives would have cost $3300 (plus the chassis), and already out of the husk probably 1.5x more. So let's say $5500 total even back then. (I built my current array around the same time)
Funny, these days you could do it with 4 spindles (4x24TB raidz2) but with the rampant price inflation in storage, you're still looking at $4500.
A whopping 150GB!
I think they just put that stuff in so they could shill their affiliate marketing articles rather than out of any actual concern.
The 3-2-1 includes the production data itself.
You have one copy of data (the data that needs to be backed up), and you have two additional copies of that data (the "3" in the strategy), where one of the backup copies is stored offsite (the "1" part), which naturally follows that it satisfies the middle "2" also.
For over a decade we used 4-3-2 strategy, but it got expensive fast and due to ever evolving budget cuts, our (local gov) customers weren't ready to pay the extra.
There's also 3-2-1-1-0, but that's just a fancy way of saying you _really_ should be doing those restore tests you never seem to have the time, because ransomware can and _will_ corrupt your backup copies.
edit: the absolute minimal setup that still satisfies the 3-2-1 rule: 1) you have your data in production directory on local NVMe stick inside your computer, 2) you copy the original production data to another directory inside the very same NVMe, 3) you occasionally manually copy the production directory data to USB and store that USB inside the shed in your backyard. - That's it. Now your home data is covered by the coveted "3-2-1" enterprise ready backup strategy.
Folio Photonics has just got some funding and they are saying they will be able to launch with capacities of 10 TB per cartridge.
ODA was never better than using LTO.
[0] https://pro.sony/en_GB/press/optical-disc-archive-generation...
This smacks of saving pennies on IT to spend dollars on lawyers.
I keep photos on Google and MS but they are short term storage only and never the only copy.
People like me who no longer pay attention enough after seeing so much upheaval in the cloud space are losing out on ideas of best practice and giving in to whatever works for the time being.
Full headline: PBS broadcaster loses access to 50TB of data comprising 70 years of TV history after contracted cloud storage vendor goes defunct — public TV channel sues Iron Mountain data center, which hosts archival materials, to ensure preservation
The channel’s attorney also reached out to Iron Mountain to inform the company of the pending case in Colorado; Iron Mountain finally admitted to holding the data. Iron Mountain initially agreed to turn over Nine PBS’ data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.
This does make sense. I'm guessing the first request some rep just went sure its you're data why not, but in reality, it's OSS's data, and you paid OSS to store it. It's also likely encrypted in someway.
TLDR still, is especially when it's a reasonable amount of data, just have more than 1 backup.
Likewise if I store data with a cloud provider. Sure, Iron Mountain shouldn't just offer it up without proof, but if I give them notice of a legal case they should be sure to hold on to it, and if I win the case they need to hand it over to me. All I need to do is provide proof that I am the rightful owner of those goods.
You are a subtenant. The tenant went bust. The case law on this is settled. Depending on state, you either have a period of time to vacate the premises or you can/do become a tenant of the property owner. People have been subleasing rooms/apartments for hundreds of years, so every possible scenario you could imagine in this space has been litigated to death.
That is real estate.
American case law about data stored on computers is that whosoever owns that computer owns all the data on that computer. Which is why US privacy law is far weaker than EU privacy law (where you own the data about yourself). I think pets.com was one of the very few exceptions where the TOS said "we won't sell your data" and the bankruptcy liquidator said "it is an asset, we sure will sell it" and the courts said "no you can't sell it".
Also your storage company case is actually it's own can of worms, see the Lego drama from a couple months ago about that. It seems obvious, but in the eyes of the law there's more paperwork than you'd think.
That is to say - I really doubt anything about this is remotely simple or straightforward, but I'm sure it's all lucrative for the lawyers involved.
This looks like a similar formality of following the process so that a court legitimises the transfer of data back to Nine PBS.
I agree that IM is probably inside of their contractual rights, but they're still handling this in a fashion that risks causing severe issues for Nine PBS. The presence of the court order seems like sufficient inducement to at least hold the deletion until the court decides, and I'd be very curious what kinds of legal remedies Nine PBS might be entitled to should IM not do so and the court decide in their favor.
They weren't recording in 4K last century.
One hour of 1970's-grade NTSC video is around 300MB.
More to the point, it's not like they archived 70 continuous years of video. Many local PBS stations produce an hour or less of video each week. This being a St. Louis station, however, it's probably somewhat more.
Like most of the time, this will probably be held for some kind of extortionist legal ransom. We just love handing our stuff over to bad-faith third-parties in the name of some kind of imagined freedom. Just can't get enough of it. Public infrastructure, national archives, city services, we just seem to be completely deer-in-the-headlights when they pull this stuff on the daily...
So here we go again. We blindly trust in private parties always magically acting in good faith when there's billions on the table incentivizing them not to.
We've been doing it like this for 40 years and it hasn't worked, but I'm sure our ideological commitments will bend reality eventually.
This is your money btw.
County libraries, city hall records, large public universities.
It should have been like "hey, UNC Chapel Hill" or "Los Angeles County" take care of this.
I mean maybe things like that happen, like UTexas just loses multiple shelves mysteriously in their special collections, but I've never heard of it.
I just hate when ideological commitments shove reasonable solutions like this off the table
At that point, why wouldn't you pick a major cloud vendor? This feels like more of a governance failure.
> This unidentified vendor provided “hardware, software and cloud-storage services” for storing the public broadcaster’s archival materials and other data.
I know there would be cheaper hardware options available (I'm partial to older surplus enterprise hardware myself) but it isn't particularly reasonable to expect a small business to choose a more DIY (at least in terms of software configuration) option. You could step down to 10TB drives, but it seems unlikely they would have wanted to risk that not being quite enough.
On top of that, since 2019 is 7 years ago, it is reasonable to expect they would have had a hardware refresh at some point, I would think 5 years would be more reasonable than 3 year for something like this. At that point, the nas would be more ($1300) but the drives would be less (8 * $300), so add either another $3700 (double if both are replaced, but maybe they figured the offsite backup could be stretched even longer than 5 years)
So, rather than $2500, it would probably have been more like $8500 + OSS or $13,300 without OSS. Still that would have meant a proper are archive and a proper backup and they wouldn't be in this mess, so I completely agree with you about what they should have done, but think you were underselling the price a good bit.
https://www.youtube.com/watch?v=lMlOkQdMQ0w
I work in several labs where I have to give this advice, which some labs are smart enough to make an ersatz backup that we store somewhere (PI's house) as a cheap offsite. Even a partial recovery is great in the face of fire/flood/explosion, or the most common error: IT fucks it up and then says "oh well".
Hope someone has some copies and they can rebuild, what a nightmare.
>This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.
Look, I know that a certain percentage of Tom's Hardware articles are just sales pitches for a NAS, but this is asinine. Iron Mountain deals largely in tape storage. 50TB of tapes fits in one of their small archive boxes. If they're using the latest tech (LTO-9), it fits on 3 tapes.
This is exactly how you are supposed to store archival media for decades, not on a NAS. A NAS is for storing the files you need to actively access.
Here is the source article...
https://current.org/2026/08/nine-pbs-sues-iron-mountain-over...
"Nine PBS" seems to be <https://www.ninepbs.org/>, A/K/A KETC, St. Louis, MO.
<https://en.wikipedia.org/wiki/KETC>
The upstream cited article which should be subbed in for this submission does clearly identify the station in the first line of its report: <https://current.org/2026/08/nine-pbs-sues-iron-mountain-over...>.
(I've submitted this to HN's mods via email. The original submission was to a very poor TomsHardware link.)
OT: The tendency of local broadcasters to fail to identify either on websites OR QUITE OFTEN ON BROADCASTS THEMSELVES even vaguely where they operate is ... a long-standing annoyance of mine. Sure, local audiences may know, but those passing through, or catching the signal at a distance, likely won't.
I've spent more than a few nights at higher elevations (campgrounds, highways) listening to clear channel stations reporting on weather and traffic "in the local area", sometimes for hours, without ever hearing what "the local area" is. That for signals travelling hundreds of km, and not infrequently multiple thousands of km.
(Don't get me started on entire countries where stations are identified simply by number, e.g., "Channel 4" in the UK.)
I worked at WCCO in Minneapolis briefly a few years ago. Right around that time, CBS corporate was attempting to standardize all their stations and associated websites. In their mind, that meant making everything "CBS News [Location]" and stripping just about everything else from the public marketing.
WCCO, KPIX in Los Angeles, and a few other stations battled back about this because the local name recognition of the call sign was stronger than CBS would be by itself.
So now, when you bop around the "Local News" section of the CBS Website, some cities have logos with the call letters, and some do not.
In short, when you see a station that that's difficult to identify, try finding out who owns it. Very often the ownership is to blame for one reason or another.
What I distinctly recall was that it was possible to get through a full weather or traffic report without the region being identified.
These days, a frequent gripe is that local-media websites (radio and television) fail to specify the state. Where a call-sign is available, I'm obliged to look that up on Wikipedia to figure out what location the story concerns. Place names are often given only as small town or county names, neither of which typically have national recognition, and many of which repeat from state to state.