tag:blogger.com,1999:blog-4503292949532760618.post5029508506009896120..comments2024-03-28T13:39:27.601-07:00Comments on DSHR's Blog: Paying for Long-Term StorageDavid.http://www.blogger.com/profile/14498131502038331594noreply@blogger.comBlogger9125tag:blogger.com,1999:blog-4503292949532760618.post-54189001010260183322014-03-04T03:54:47.993-08:002014-03-04T03:54:47.993-08:00David, please forgive me for commenting on an old ...David, please forgive me for commenting on an old post. My math skills are somewhat outdated, but it seems to me that the formula on slide 16 of Serge Goldstein's presentation that you cite is mistaken: He holds that the series converges on I/(1-d)^r, but I think the equation should read <b>I/(1-(1-d)^r)</b>, according to the formula for geometric series. Please correct me if I'm wrong, but at least this way the results make sense.Unknownhttps://www.blogger.com/profile/09541961062161515260noreply@blogger.comtag:blogger.com,1999:blog-4503292949532760618.post-16704226170821583012011-05-30T11:30:12.595-07:002011-05-30T11:30:12.595-07:00Anyone thinking about endowment models for storage...Anyone thinking about endowment models for storage should read the recent speech by Andrew Haldane and Richard Davies of the Bank of England, which <a href="http://www.bankofengland.co.uk/publications/speeches/2011/speech495.pdf" rel="nofollow">shows that decision makers systematically and increasingly use unrealistically high discount rates</a>. This means that the marketing headwind endowment suffers is even greater than the raw numbers suggest.David.https://www.blogger.com/profile/14498131502038331594noreply@blogger.comtag:blogger.com,1999:blog-4503292949532760618.post-85112928711483678042011-05-11T14:25:18.968-07:002011-05-11T14:25:18.968-07:00The Library of Congress is concerned with the pric...The Library of Congress is concerned with the price they are paying for disk storage, and have instituted a policy that limits the cost of new storage systems (media, enclosure, switches, ...) to 10x the cost of raw consumer disk storage.David.https://www.blogger.com/profile/14498131502038331594noreply@blogger.comtag:blogger.com,1999:blog-4503292949532760618.post-83463530719716277072011-05-11T11:20:27.881-07:002011-05-11T11:20:27.881-07:00USC is charging $70/TB/month for a copy on disk an...USC is charging $70/TB/month for a copy on disk and $1K/TB for a copy on tape for 20 years both for internal and external customers, with no bandwidth charges.David.https://www.blogger.com/profile/14498131502038331594noreply@blogger.comtag:blogger.com,1999:blog-4503292949532760618.post-45637679081733625722011-03-02T10:45:48.354-08:002011-03-02T10:45:48.354-08:00David,
You've hit the nail on the head. If D...David,<br /><br />You've hit the nail on the head. If DataSpace fails, it will be because faculty are unwilling to pay even a one-time charge if they perceive that charge to be unreasonably high. Because the granting agencies require a data plan, what I suspect will happen is something along the lines of:<br /><br />a) Faculty will ask their departmental sysadmins to write the data management plan.<br />b) The departmental sysadmins, most of whom are graduate students, will say "hey, we can do this for a whole lot less than those DataSpace folks are charging", and will go out and buy the $100 Best Buy terabyte drive.<br />c) The plan will be submitted (it will say "reliable departmental servers", not "cheap Best Buy drives") and the reviewers will say, that's fine, that solution should last a few years, which is all that's needed anyway.<br /><br />If people really want "indefinite", reliable, accessible storage, then $100 Best Buy drives won't do it. On the other hand, I agree that we are charging too much. I am cautiously optimistic that we can architect a storage solution here that is a lot cheaper than what we currently have for use with DataSpace.<br /><br />SergeSergehttps://www.blogger.com/profile/03441392651096692358noreply@blogger.comtag:blogger.com,1999:blog-4503292949532760618.post-7018574517187303382011-03-02T09:03:09.178-08:002011-03-02T09:03:09.178-08:00Thank you, Serge. I agree that you can "rob P...Thank you, Serge. I agree that you can "rob Peter to pay Paul" in the long term but if costs flatten in the first few years you do need a substantial margin over your initial costs to be safe, as my reply to Chris points out.<br /><br />And in order to "rob Peter to pay Paul" you need a continuing flow of new customers, which I argue is going to be a hard sell. I am not computing the 3000% margin against USB disks, but against the one-off cost of bare 3.5" 2TB SATA drives at retail. These are the same drives that large data centers use for bulk storage. Clearly, there are other costs on top, for the servers, switches, bandwidth, power and cooling. I point that out. But the marketing problem remains. The perception among your potential customers is that this endowed storage is 3000% more expensive than the off-the-shelf storage they can buy themselves. Do they perceive the added value of long-term storage as worth a factor of 30x in cost?David.https://www.blogger.com/profile/14498131502038331594noreply@blogger.comtag:blogger.com,1999:blog-4503292949532760618.post-78725814347567335332011-03-02T08:15:23.417-08:002011-03-02T08:15:23.417-08:00David,
Thanks for a very insightful and useful a...David,<br /> <br />Thanks for a very insightful and useful analysis. I wanted to comment on a couple of issues that you raise, and how the model might address them.<br /> <br />You point out that, while it is true that the total cost of storage has been declining, such a decline may not be sustained, and if storage costs stay flat for a few years, then the endowment model will fail. I am hesitant to bet against technology, and the declines in storage costs (all costs, not just disk drives) have been so steady and dramatic over the past 40 years, that I think basing a model on this assumption is a pretty safe bet. However, the model does have a way to address changes in the rate of decrease of storage costs. The cost factor the model charges can be adjusted from year to year. Does this involve robing Peter to pay Paul? Yes, in a way it does, but this is how banks and many other organizations operate. (The 4% 40-year fixed term mortgage can end up looking bad when interest rates rise above 4%, but the bank makes up for this by charging higher rates on other loans). As long as storage costs decline over the long haul, and we continue to attract new customers (people keep doing research that generates data which needs to be saved), then the model does have a mechanism for adjusting to "blips" in storage costs.<br /> <br />The second area where I think some comment is needed has to do with the 3000% surcharge we are imposing on our customers. I believe this is a much more serious problem than the possibility of flat storage costs. Your analysis assumes that we are doing this to provide us with a "buffer" against blips in storage costs. In fact, we are charging our customers exactly 2x what we are paying ourselves for disk space. The problem here is that University data centers are not designed to provide low-cost long-term storage. Our sysadmins do not buy USB disk drives at Best Buy. They buy enterprise-level storage, designed to sustain high use and high reliability. I would love to find a supplier who would provide us with reliable, scalable storage at $100/terabyte (and no usage or bandwidth fees), but that hasn't happened. Maybe someone in the research community will create such a center. I think there is a business opportunity here for an enterprising group of graduate students.<br /><br />Serge Goldstein, OITSergehttps://www.blogger.com/profile/03441392651096692358noreply@blogger.comtag:blogger.com,1999:blog-4503292949532760618.post-58848380264228989932011-02-28T17:32:10.948-08:002011-02-28T17:32:10.948-08:00Strictly, the graph shows the history of the capac...Strictly, the graph shows the history of the capacity of raw hard disk, which certainly does not increase by 10 every year. Kryder's law actually says that the areal density on the platters doubles every year, but as of recent years this hasn't been achieved - if it had been we would have 4TB drives by now. So the flattening of the curve is not a projection, but a fact. The question is how long will it last?<br /><br />More importantly, as I point out in the talk it isn't the raw cost of the disk but the total cost of storage that determines how much you need to pay to endow data. The total cost has been dropping, but not as fast as the cost of the raw disk. I discuss the reasons for this in the talk.<br /><br />The question Chris raises is the effect of a blip. Lets use Serge Goldstein's numbers as an example. He projects total storage costs dropping 20% a year, and a 4-year life for the hardware. Thus for every $2 he charges, he spends $1 on storage and keeps $1 for 4 years. Then he spends about $0.40 on new storage and keeps $0.60 for 4 years.<br /><br />Suppose the total cost of storage stays constant for the first 3 years, then starts dropping as expected 20% a year. Now Serge spends $1 and keeps $1 for 4 years. At this point it costs him $0.80 to replace the storage, leaving him $0.20 for the next 4 years. But at the end of those 4 years he can't afford to pay the $0.32 that replacing the storage will cost him.<br /><br />Serge is right that the cost series converges, but it only does so if a non-zero proportion of the years see a drop in cost. However, his computation that twice the cost of storage is enough is true only if <i>all</i> years see a 20% drop in cost. Even a few years with flat costs somewhere in the future will cause the value to which the series converges to exceed the $2, and the endowed data to run out of money.<br /><br />Some endowed data proposals charge higher margins that Princeton's 3000%, and some charge less. But margins of this magnitude or even greater are needed to insure against the possibility of the curve flattening. The alternative would be a futures market in storage, so that trusts endowing data could hedge against the volatility. We've seen how well derivatives of this kind work when the unexpected happens.David.https://www.blogger.com/profile/14498131502038331594noreply@blogger.comtag:blogger.com,1999:blog-4503292949532760618.post-89869532408316854822011-02-28T00:30:20.968-08:002011-02-28T00:30:20.968-08:00Thanks David. I hadn't heard the name of the s...Thanks David. I hadn't heard the name of the storage law, but Bryan Lawrence first alerted me to its effects. You suggest the exponential growth may slow down, citing higher costs of 2.5" drives and Flash; but the Kryder graph you included shows easy factors of 10 for each time point, so surely this is just another generational blip?Chris Rusbridgehttps://www.blogger.com/profile/06087447503626434385noreply@blogger.com