Rules of thumb in data engineering

Gray, Jim; Shenoy, Prashant

doi:10.1109/icde.2000.839382

Cited by 133 publications

(73 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Because of high latency of disk, larger clusters are generally expected to yield better sort performance despite the limited fan-out in run generation and the increased number of merge steps. In fact, it is claimed that the optimal size of cluster has steadily increased roughly from 16 or 32 KBytes to 128 KBytes or even larger over the past decade, as the gap between latency and bandwidth improvement has become wider [7,9].…”

Section: External Sortmentioning

confidence: 99%

A case for flash memory ssd in enterprise database applications

Lee

Moon

Park

et al. 2008

Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data

234

119

View full text Add to dashboard Cite

Due to its superiority such as low access latency, low energy consumption, light weight, and shock resistance, the success of flash memory as a storage alternative for mobile computing devices has been steadily expanded into personal computer and enterprise server markets with ever increasing capacity of its storage. However, since flash memory exhibits poor performance for small-to-moderate sized writes requested in a random order, existing database systems may not be able to take full advantage of flash memory without elaborate flash-aware data structures and algorithms. The objective of this work is to understand the applicability and potential impact that flash memory SSD (Solid State Drive) has for certain type of storage spaces of a database server where sequential writes and random reads are prevalent. We show empirically that up to more than an order of magnitude improvement can be achieved in transaction processing by replacing magnetic disk with flash memory SSD for transaction log, rollback segments, and temporary table spaces.

show abstract

Section: External Sortmentioning

confidence: 99%

A case for flash memory ssd in enterprise database applications

Lee

Moon

Park

et al. 2008

Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data

234

119

View full text Add to dashboard Cite

show abstract

“…Data format transformation is needed because, in our case, CDR is not ideal to be directly processed by MapReduce. Based on the works [1,9], in fixed disk mode, sequential access has much better performance than random access. So in our case of using conventional disks, sequential pattern implementation will improve ingress performance.…”

Section: System Analysismentioning

confidence: 99%

High Performance CDR Processing with MapReduce

Agung

Kistijantoro

2016

J. ICT Res. Appl.

View full text Add to dashboard Cite

Abstract.A call detail record (CDR) is a data record produced by telecommunication equipment consisting of call detail transaction logs. It contains valuable information for many purposes in several domains, such as billing, fraud detection and analytical purposes. However, in the real world these needs face a big data challenge. Billions of CDRs are generated every day and the processing systems are expected to deliver results in a timely manner. The capacity of our current production system is not enough to meet these needs. Therefore a better performing system based on MapReduce and running on Hadoop cluster was designed and implemented. This paper presents an analysis of the previous system and the design and implementation of the new system, called MS2. In this paper also empirical evidence is provided to demonstrate the efficiency and linearity of MS2. Tests have shown that MS2 reduces overhead by 44% and speeds up performance nearly twice compared to the previous system. From benchmarking with several related technologies in large-scale data processing, MS2 was also shown to perform better in the case of CDR batch processing. When it runs on a cluster consisting of eight CPU cores and two conventional disks, MS2 is able to process 67,000 CDRs/second.

show abstract

“…Following Moore's law, the speed of hardware is likely to increase by almost a factor of two in less than two years. However, as processor speeds continue to outpace memory speeds [10], the gap between processor and memory performance increases by about 50 % per year [5]. Thus, there is a need to minimize the effects of such high performance hardware.…”

Section: Preliminariesmentioning

confidence: 99%

Rig: A Simple, Secure and Flexible Design for Password Hashing

Chang

Jati

Mishra

et al. 2015

Information Security and Cryptology

View full text Add to dashboard Cite

Abstract. Password Hashing, a technique commonly implemented by a server to protect passwords of clients, by performing a one-way transformation on the password, turning it into another string called the hashed password. In this paper, we introduce a secure password hashing framework Rig which is based on secure cryptographic hash functions. It provides the flexibility to choose different functions for different phases of the construction. The design of the scheme is very simple to implement in software and is flexible as the memory parameter is independent of time parameter (no actual time and memory trade-off) and is strictly sequential (difficult to parallelize) with comparatively huge memory consumption that provides strong resistance against attackers using multiple processing units. It supports client-independent updates, i.e., the server can increase the security parameters by updating the existing password hashes without knowing the password. Rig can also support the server relief protocol where the client bears the maximum effort to compute the password hash, while there is minimal effort at the server side. We analyze Rig and show that our proposal provides an exponential time complexity against the low-memory attack.

show abstract

Rules of thumb in data engineering

Cited by 133 publications

References 9 publications

A case for flash memory ssd in enterprise database applications

A case for flash memory ssd in enterprise database applications

High Performance CDR Processing with MapReduce

Rig: A Simple, Secure and Flexible Design for Password Hashing

Contact Info

Product

Resources

About