ColdStore: A Storage System for Archival Data

Chi, Mucong; Li, Jun; Yang, Jie

doi:10.1007/s11277-019-06989-5

Cited by 2 publications

(2 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this regard, the namenode jobs are: i) check if there is a free memory for the new dataset, ii) provide the free datablock IDs to the new dataset, and iii) splitting the large files =>64 MB to assigning to multi datablocks. After all operations above, the dataset will continue to the assigned datablock to be stored in there and send the required metadata files to the namenode to update the namenode on the new dataset files and location inside HDFS [9], [10].…”

Section: Introductionmentioning

confidence: 99%

Tiny datablock in saving Hadoop distributed file system wasted memory

Al-Masadeh

Azmi

Ahmad

2023

IJECE

View full text Add to dashboard Cite

<p>Hadoop distributed file system (HDFS) is the file system whereby Hadoop is use it to store all the upcoming data inside it. Since it been declared, HDFS is consuming a huge memory amount in order to serve a normal dataset. Nonetheless, the current file saving mechanism in HDFS save only one file in one datablock. Thus, a file with just 5 Mb in size will take up the whole datablock capacity causing the rest of the memory unavailable for other upcoming files, and this is considered a huge waste of memory in serving a normal size dataset. This paper proposed a method called tiny datablock-HDFS (TD-HDFS) to increase the usability of HDFS memory and increase the file hosting capabilities by reducing the datablock size to the minimum capacity, and then merging all the related datablocks into one master datablock. This master datablock consists of tiny virtual datablocks that contain the related small files together; will exploit the full memory of the master datablock. The result of this study is a running HDFS with a minimum amount of wasted memory with the same read/write data performance. The results were examined through a comparison between the standard HDFS file hosting and the proposed solution of this study.</p><textarea id="BFI_DATA" style="width: 1px; height: 1px; display: none;"></textarea><textarea id="BFI_DATA" style="width: 1px; height: 1px; display: none;"></textarea><textarea id="BFI_DATA" style="width: 1px; height: 1px; display: none;"></textarea><textarea id="BFI_DATA" style="width: 1px; height: 1px; display: none;"></textarea><div id="WidgetFloaterPanels" class="LTRStyle" style="display: none; text-align: left; direction: ltr; visibility: hidden;"><div id="WidgetFloater" style="display: none;" onmouseover="Microsoft.Translator.OnMouseOverFloater()" onmouseout="Microsoft.Translator.OnMouseOutFloater()"><div id="WidgetLogoPanel"><span id="WidgetTranslateWithSpan"><span>TRANSLATE with </span><img id="FloaterLogo" alt="" /></span> <span id="WidgetCloseButton" title="Exit Translation" onclick="Microsoft.Translator.FloaterOnClose()">x</span></div><div id="LanguageMenuPanel"><div class="DDStyle_outer"><input id="LanguageMenu_svid" style="display: none;" onclick="this.select()" type="text" name="LanguageMenu_svid" value="en" /> <input id="LanguageMenu_textid" style="display: none;" onclick="this.select()" type="text" name="LanguageMenu_textid" /> <span id="__LanguageMenu_header" class="DDStyle" onclick="return LanguageMenu && !LanguageMenu.Show('__LanguageMenu_popup', event);" onkeydown="return LanguageMenu && !LanguageMenu.Show('__LanguageMenu_popup', event);">English</span><div style="position: relative; text-align: left; left: 0;"><div style="position: absolute; ;left: 0px;"><div id="__LanguageMenu_popup" class="DDStyle" style="display: none;"><table id="LanguageMenu" border="0"><tbody><tr><td><a onclick="return LanguageMenu.onclick('ar');" tabindex="-1" href="#ar">Arabic</a></td><td><a onclick="return LanguageMenu.onclick('he');" tabindex="-1" href="#he">Hebrew</a></td><td><a onclick="return LanguageMenu.onclick('pl');" tabindex="-1" href="#pl">Polish</a></td></tr><tr><td><a onclick="return LanguageMenu.onclick('bg');" tabindex="-1" href="#bg">Bulgarian</a></td><td><a onclick="return LanguageMenu.onclick('hi');" tabindex="-1" href="#hi">Hindi</a></td><td><a onclick="return LanguageMenu.onclick('pt');" tabindex="-1" href="#pt">Portuguese</a></td></tr><tr><td><a onclick="return LanguageMenu.onclick('ca');" tabindex="-1" href="#ca">Catalan</a></td><td><a onclick="return LanguageMenu.onclick('mww');" tabindex="-1" href="#mww">Hmong Daw</a></td><td><a onclick="return LanguageMenu.onclick('ro');" tabindex="-1" href="#ro">Romanian</a></td></tr><tr><td><a onclick="return LanguageMenu.onclick('zh-CHS');" tabindex="-1" href="#zh-CHS">Chinese Simplified</a></td><td><a onclick="return LanguageMenu.onclick('hu');" tabindex="-1" href="#hu">Hungarian</a></td><td><a onclick="return LanguageMenu.onclick('ru');" tabindex="-1" href="#ru">Russian</a></td></tr><tr><td><a onclick="return LanguageMenu.onclick('zh-CHT');" tabindex="-1" href="#zh-CHT">Chinese Traditional</a></td><td><a onclick="return LanguageMenu.onclick('id');" tabindex="-1" href="#id">Indonesian</a></td><td><a onclick="return LanguageMenu.onclick('sk');" tabindex="-1" href="#sk">Slovak</a></td></tr><tr><td><a onclick="return LanguageMenu.onclick('cs');" tabindex="-1" href="#cs">Czech</a></td><td><a onclick="return LanguageMenu.onclick('it');" tabindex="-1" href="#it">Italian</a></td><td><a onclick="return LanguageMenu.onclick('sl');" tabindex="-1" href="#sl">Slovenian</a></td></tr><tr><td><a onclick="return LanguageMenu.onclick('da');" tabindex="-1" href="#da">Danish</a></td><td><a onclick="return LanguageMenu.onclick('ja');" tabindex="-1" href="#ja">Japanese</a></td><td><a onclick="return LanguageMenu.onclick('es');" tabindex="-1" href="#es">Spanish</a></td></tr><tr><td><a onclick="return LanguageMenu.onclick('nl');" tabindex="-1" href="#nl">Dutch</a></td><td><a onclick="return LanguageMenu.onclick('tlh');" tabindex="-1" href="#tlh">Klingon</a></td><td><a onclick="return LanguageMenu.onclick('sv');" tabindex="-1" href="#sv">Swedish</a></td></tr><tr><td><a onclick="return LanguageMenu.onclick('en');" tabindex="-1" href="#en">English</a></td><td><a onclick="return LanguageMenu.onclick('ko');" tabindex="-1" href="#ko">Korean</a></td><td><a onclick="return LanguageMenu.onclick('th');" tabindex="-1" href="#th">Thai</a></td></tr><tr><td><a onclick="return LanguageMenu.onclick('et');" tabindex="-1" href="#et">Estonian</a></td><td><a onclick="return LanguageMenu.onclick('lv');" tabindex="-1" href="#lv">Latvian</a></td><td><a onclick="return LanguageMenu.onclick('tr');" tabindex="-1" href="#tr">Turkish</a></td></tr><tr><td><a onclick="return LanguageMenu.onclick('fi');" tabindex="-1" href="#fi">Finnish</a></td><td><a onclick="return LanguageMenu.onclick('lt');" tabindex="-1" href="#lt">Lithuanian</a></td><td><a onclick="return LanguageMenu.onclick('uk');" tabindex="-1" href="#uk">Ukrainian</a></td></tr><tr><td><a onclick="return LanguageMenu.onclick('fr');" tabindex="-1" href="#fr">French</a></td><td><a onclick="return LanguageMenu.onclick('ms');" tabindex="-1" href="#ms">Malay</a></td><td><a onclick="return LanguageMenu.onclick('ur');" tabindex="-1" href="#ur">Urdu</a></td></tr><tr><td><a onclick="return LanguageMenu.onclick('de');" tabindex="-1" href="#de">German</a></td><td><a onclick="return LanguageMenu.onclick('mt');" tabindex="-1" href="#mt">Maltese</a></td><td><a onclick="return LanguageMenu.onclick('vi');" tabindex="-1" href="#vi">Vietnamese</a></td></tr><tr><td><a onclick="return LanguageMenu.onclick('el');" tabindex="-1" href="#el">Greek</a></td><td><a onclick="return LanguageMenu.onclick('no');" tabindex="-1" href="#no">Norwegian</a></td><td><a onclick="return LanguageMenu.onclick('cy');" tabindex="-1" href="#cy">Welsh</a></td></tr><tr><td><a onclick="return LanguageMenu.onclick('ht');" tabindex="-1" href="#ht">Haitian Creole</a></td><td><a onclick="return LanguageMenu.onclick('fa');" tabindex="-1" href="#fa">Persian</a></td><td> </td></tr></tbody></table><img style="height: 7px; width: 17px; border-width: 0px; left: 20px;" alt="" /></div></div></div></div><script type="text/javascript">// <![CDATA[ var LanguageMenu; var LanguageMenu_keys=["ar","bg","ca","zh-CHS","zh-CHT","cs","da","nl","en","et","fi","fr","de","el","ht","he","hi","mww","hu","id","it","ja","tlh","ko","lv","lt","ms","mt","no","fa","pl","pt","ro","ru","sk","sl","es","sv","th","tr","uk","ur","vi","cy"]; var LanguageMenu_values=["Arabic","Bulgarian","Catalan","Chinese Simplified","Chinese Traditional","Czech","Danish","Dutch","English","Estonian","Finnish","French","German","Greek","Haitian Creole","Hebrew","Hindi","Hmong Daw","Hungarian","Indonesian","Italian","Japanese","Klingon","Korean","Latvian","Lithuanian","Malay","Maltese","Norwegian","Persian","Polish","Portuguese","Romanian","Russian","Slovak","Slovenian","Spanish","Swedish","Thai","Turkish","Ukrainian","Urdu","Vietnamese","Welsh"]; var LanguageMenu_callback=function(){ }; var LanguageMenu_popupid='__LanguageMenu_popup'; // ]]></script></div><div id="CTFLinksPanel"><span id="ExternalLinksPanel"><a id="HelpLink" title="Help" href="https://go.microsoft.com/?linkid=9722454" target="_blank"> <img id="HelpImg" alt="" /></a> <a id="EmbedLink" title="Get this widget for your own site" href="javascript:Microsoft.Translator.FloaterShowEmbed()"> <img id="EmbedImg" alt="" /></a> <a id="ShareLink" title="Share translated page with friends" href="javascript:Microsoft.Translator.FloaterShowSharePanel()"> <img id="ShareImg" alt="" /></a> </span></div><div id="FloaterProgressBar"> </div></div><div id="WidgetFloaterCollapsed" style="display: none;" onmouseover="Microsoft.Translator.OnMouseOverFloater()"><span>TRANSLATE with </span><img id="CollapsedLogoImg" alt="" /></div><div id="FloaterSharePanel" style="display: none;"><div id="ShareTextDiv"><span id="ShareTextSpan"> COPY THE URL BELOW </span></div><div id="ShareTextboxDiv"><input id="ShareTextbox" onclick="this.select()" type="text" name="ShareTextbox" readonly="readonly" />  <a id="EmailLink" title="Email this translation"> <img id="EmailImg" alt="" /></a></div><div id="ShareFooter"><span id="ShareHelpSpan"><a id="ShareHelpLink"> <img id="ShareHelpImg" alt="" /></a></span> <span id="ShareBackSpan"><a id="ShareBack" title="Back To Translation" href="javascript:Microsoft.Translator.FloaterOnShareBackClick()"> Back</a></span></div><input id="EmailSubject" type="hidden" name="EmailSubject" value="Check out this page in {0} translated from {1}" /> <input id="EmailBody" type="hidden" name="EmailBody" value="Translated: {0}%0d%0aOriginal: {1}%0d%0a%0d%0aAutomatic translation powered by Microsoft® Translator%0d%0ahttp://www.bing.com/translator?ref=MSTWidget" /> <input id="ShareHelpText" type="hidden" value="This link allows visitors to launch this page and automatically translate it to {0}." /></div><div id="FloaterEmbed" style="display: none;"><div id="EmbedTextDiv"><span id="EmbedTextSpan">EMBED THE SNIPPET BELOW IN YOUR SITE</span> <a id="EmbedHelpLink" title="Copy this code and place it into your HTML."> <img id="EmbedHelpImg" alt="" /></a></div><div id="EmbedTextboxDiv"><input id="EmbedSnippetTextBox" onclick="this.select()" type="text" name="EmbedSnippetTextBox" value="<div id='MicrosoftTranslatorWidget' class='Dark' style='color:white;background-color:#555555'></div><script type='text/javascript'>setTimeout(function(){var s=document.createElement('script');s.type='text/javascript';s.charset='UTF-8';s.src=((location && location.href && location.href.indexOf('https') == 0)?'https://ssl.microsofttranslator.com':'http://www.microsofttranslator.com')+'/ajax/v3/WidgetV3.ashx?siteData=ueOIGRSKkd965FeEGM5JtQ**&ctf=true&ui=true&settings=manual&from=en';var p=document.getElementsByTagName('head')[0]||document.documentElement;p.insertBefore(s,p.firstChild); },0);</script>" readonly="readonly" /></div><div id="EmbedNoticeDiv"><span id="EmbedNoticeSpan">Enable collaborative features and customize widget: <a href="http://www.bing.com/widget/translator" target="_blank">Bing Webmaster Portal</a></span></div><div id="EmbedFooterDiv"><span id="EmbedBackSpan"><a title="Back To Translation" href="javascript:Microsoft.Translator.FloaterOnEmbedBackClick()">Back</a></span></div></div><script type="text/javascript">// <![CDATA[ var intervalId = setInterval(function () { if (MtPopUpList) { LanguageMenu = new MtPopUpList(); var langMenu = document.getElementById(LanguageMenu_popupid); var origLangDiv = document.createElement("div"); origLangDiv.id = "OriginalLanguageDiv"; origLangDiv.innerHTML = "<span id='OriginalTextSpan'>ORIGINAL: </span><span id='OriginalLanguageSpan'></span>"; langMenu.appendChild(origLangDiv); LanguageMenu.Init('LanguageMenu', LanguageMenu_keys, LanguageMenu_values, LanguageMenu_callback, LanguageMenu_popupid); window["LanguageMenu"] = LanguageMenu; clearInterval(intervalId); } }, 1); // ]]></script></div>

show abstract

Section: Introductionmentioning

confidence: 99%

Tiny datablock in saving Hadoop distributed file system wasted memory

Al-Masadeh

Azmi

Ahmad

2023

IJECE

View full text Add to dashboard Cite

show abstract

“…How to improve the query performance of XML database is particularly important under the premise of massive data. XML file is essentially a structured carrier for storing information, and its processing capacity for its own database is limited [2]. The advantage is that it is semistructured and can customize the schema.…”

Section: Introductionmentioning

confidence: 99%

Wireless Network Communication in the XML Metadata Storage of Wushu Historical Archives

Xin

2021

Wireless Communications and Mobile Computing

View full text Add to dashboard Cite

The wireless communication network has a huge promotion effect on data processing. As an important standard for internet data transmission, XML markup language is currently widely used in the internet. The rapid development of XML has brought fresh blood to the research of database. This article is aimed at studying the application of wireless network communication in the storage of the XML metadata database of Wushu Historical Archives. One part uses data load-balancing algorithms and two-way path constraint algorithms to conduct research on the metadata storage of XML databases, such as document size, document loading, and document query. Combining wireless technology and comparing with SGML and HTML, it highlights the flexibility and importance of XML database. The other part analyzes the XML database of Wushu Historical Archives itself and uses questionnaire surveys combined with the experience of 30 users randomly checked to provide constructive suggestions for the construction of XML database, such as strengthening the operation page design, text, and graphics. The experimental results show that the time consumed by the XML database in the case of bidirectional path indexing is 0.37 s, and the time consumed in the case of unidirectional path indexing is 0.27 s. This data shows that the application of wireless network communication technology has played an important role in the easy update and flexibility of the XML database. The data load capacity presented by the two path indexes is inconsistent; the current experience of XML database mainly focuses on the copy of the database quantity.

show abstract

ColdStore: A Storage System for Archival Data

Cited by 2 publications

References 21 publications

Tiny datablock in saving Hadoop distributed file system wasted memory

Tiny datablock in saving Hadoop distributed file system wasted memory

Wireless Network Communication in the XML Metadata Storage of Wushu Historical Archives

Contact Info

Product

Resources

About