This repository has been archived by the owner on Jan 13, 2022. It is now read-only.

Develop#38

Open

deeps123123 wants to merge 526 commits intomasterfrom develop

+150,944-27,566

This pull request is big! We're only showing the most recent 250 commits

Commits on Nov 8, 2012

Fix build SnapshotNode imports

tomasz
authored and
Alex Feinberg
committed
HDFS: fix test bug in TestTransferBlock

sdong
authored and
Alex Feinberg
committed
FsShellService: Fix bugs

sdong
authored and
Alex Feinberg
committed
Rethrow the exception

dms
authored and
Alex Feinberg
committed
Fix re-replication in BlockPlacementPolicyConfigurable.

pritam
authored and
Alex Feinberg
committed
[HDFS Build] fix build_all failure by excluding namespace-notifier build.xml

weiyan
authored and
Alex Feinberg
committed
Fixing probably-missed sync in VolumeMap.update

alushnikov
authored and
Alex Feinberg
committed
[L38-10-bump] Plugin JournalManager to hdfs-core

tomasz
authored and
Alex Feinberg
committed
[L38-11-elopt] Port edit log optimizations.

tomasz
authored and
Alex Feinberg
committed
[L38-12-bumpavatar] Plugin journal manager into avatarnodes

tomasz
authored and
Alex Feinberg
committed
Fix for the SocketTimeoutException

dms
authored and
Alex Feinberg
committed
[L38-13-fixes] Fixing some perf issues and todo's for new layout

tomasz
authored and
Alex Feinberg
committed
[L38-14-fix] Fix problem with syncing in Ingest

tomasz
authored and
Alex Feinberg
committed
[L38-15-fixes] Enable checkpointing for test case

tomasz
authored and
Alex Feinberg
committed
HDFS FsShellService: add getFileStatus() and exists()

sdong
authored and
Alex Feinberg
committed
Add the GC counters to job status Task #1358198

zhuohuang
authored and
Alex Feinberg
committed
Set fs.ha.retrywrites for AvatarBalancer.

pritam
authored and
Alex Feinberg
committed
[L38-17-fix] Fix corner case with non-clean primary shutdown

tomasz
authored and
Alex Feinberg
committed
[L38-18-metrics] Add lastTransactionId to avatar metrics

tomasz
authored and
Alex Feinberg
committed
HDFS Fsck doesn't show total number of data nodes or racks if user asks for file level of information or further

sdong
authored and
Alex Feinberg
committed
[L38-20-rolladmin] Add dfsadmin command rollEditLog

tomasz
authored and
Alex Feinberg
committed
[L38-21-merge] Fix merging bug

tomasz
authored and
Alex Feinberg
committed
Enhance FsShell to support hardlink.

pritam
authored and
Alex Feinberg
committed
Add total mapper cpu and total reducer cpu into job stats log (Hadoop Changes only)

adobriyal
authored and
Alex Feinberg
committed
Improve concurrency of the client cache

dms
authored and
Alex Feinberg
committed
Print the JVM environment before starting

dms
authored and
Alex Feinberg
committed
Update Hoopla test for more choices: local mode and disable checksum verification

sdong
authored and
Alex Feinberg
committed
HDFS AvatarDataNode to resolve DNS address on some failures

sdong
authored and
Alex Feinberg
committed
Retries in JobTrackerReporter

rvadali
authored and
Alex Feinberg
committed
Shutdown responder cleanly on avatarnode shutdown

tomasz
authored and
Alex Feinberg
committed
[L38-22-tests] Fixing test cases after merge

tomasz
authored and
Alex Feinberg
committed
Dump threads if the task doesn't progress

dms
authored and
Alex Feinberg
committed
Fix the retry in JobTrackerReporter

rvadali
authored and
Alex Feinberg
committed
Disallow a name directory to be auto-formatted on error

hkuang
authored and
Alex Feinberg
committed
Fix broken hadoop build

tomasz
authored and
Alex Feinberg
committed
[L38-23-fixes] Stop namenode proxy when initializing Standby

tomasz
authored and
Alex Feinberg
committed
Introducing a timeout for umbilical

dms
authored and
Alex Feinberg
committed
Fix timeout handling

dms
authored and
Alex Feinberg
committed
Fix the protocol getting

dms
authored and
Alex Feinberg
committed
Use PureJavaCRC32 and add a local write benchmark.

pritam
authored and
Alex Feinberg
committed
Modification of Hadoop codebase to allow corona task tracker to utilize a list of working directories from a different file

alvinl
authored and
Alex Feinberg
committed
Quick fix to DatDirFileReader

alvinl
authored and
Alex Feinberg
committed
Add aliases to the list of federation keys.

pritam
authored and
Alex Feinberg
committed
change start-corona script to call start-proxyjt.sh to first

jeanxu
authored and
Alex Feinberg
committed
Dump threads when CJT doesn't heartbeat

dms
authored and
Alex Feinberg
committed
Increaseing zookeeper connect and session timeout for unit tests.

pritam
authored and
Alex Feinberg
committed
Add datanode to outstanding list during registration.

pritam
authored and
Alex Feinberg
committed
Fix namenode formatting logic

tomasz
authored and
Alex Feinberg
committed
Log the CTT exception better

dms
authored and
Alex Feinberg
committed
[L38-24-tests] Clean up checkpointing in tests

tomasz
authored and
Alex Feinberg
committed
Randomly choose ports instead of sequentially.

pritam
authored and
Alex Feinberg
committed
[L38-25-ckpt] Add logic for readin fs.checkpoint.size

tomasz
authored and
Alex Feinberg
committed
Fix port conflicts in the tests

tomasz
authored and
Alex Feinberg
committed
Update the GC counters. 1) Only get the heap size for Old gen. 2) Add the counter to record the max aftergc heap size. 3) Fix some code format issue.

zhuohuang
authored and
Alex Feinberg
committed
FsShellService: make backlog queue configurable and add File Not Found exception

sdong
authored and
Alex Feinberg
committed
FSShellService: file didn't check in for D571836

sdong
authored and
Alex Feinberg
committed
HDFS DataNode: remove an unnecessary string format

sdong
authored and
Alex Feinberg
committed
Enhance image visitors for hardlinks.

pritam
authored and
Alex Feinberg
committed
API get list of files hardlinked to a given file.

pritam
authored and
Alex Feinberg
committed
Fix trunk build.

pritam
authored and
Alex Feinberg
committed
Revert "Randomly choose ports instead of sequentially."

tomasz
authored and
Alex Feinberg
committed
Add a HighMemoryUsageKilled counter for a Hadoop job

adobriyal
authored and
Alex Feinberg
committed
andomly choose ports for MiniDfsCluster

tomasz
authored and
Alex Feinberg
committed
Enforce that at least one image directory is present

tomasz
authored and
Alex Feinberg
committed
Add condition to check if edit log is open

tomasz
authored and
Alex Feinberg
committed
Add hardlinkId to FileStatusExtended and fix bug in totalFiles.

pritam
authored and
Alex Feinberg
committed
Fix bugs in TestAvatarCheckpointing.

pritam
authored and
Alex Feinberg
committed
Verify total number of inodes during failover.

pritam
authored and
Alex Feinberg
committed
Fix the regression caused by D349832.

hkuang
authored and
Alex Feinberg
committed
Add log4j.properties.scribeappender to VENDOR/hadoop-0.20/conf.

pritam
authored and
Alex Feinberg
committed
Another fix for TestAvatarCheckpointing.

pritam
authored and
Alex Feinberg
committed
Modification to refreshVolumes to allow for the datanode to also remove directories in addition to adding directories.

alvinl
authored and
Alex Feinberg
committed
Retry to instantiate datanodes and avataranodes for miniavatarcluster

tomasz
authored and
Alex Feinberg
committed
Remove the default value for volumes tolerated

tomasz
authored and
Alex Feinberg
committed
Add additional check for dir executable when checking directories

tomasz
authored and
Alex Feinberg
committed
Lost task tracker should not fail succeeded map tasks in map-only job

rvadali
authored and
Alex Feinberg
committed
Log the location of task logs.

rvadali
authored and
Alex Feinberg
committed
Extend NNBench to support hardlinks.

pritam
authored and
Alex Feinberg
committed
Upgrade ZooKeeper to 3.4.3
Alex Feinberg
committed
HDFS: Backward compabile SendBlock constructor

sdong
authored and
Alex Feinberg
committed
Command line option to start avatarnodes with NULL zookeeper.

pritam
authored and
Alex Feinberg
committed
Add setReuseAddress to free ports in MiniAvatarCluster

tomasz
authored and
Alex Feinberg
committed
Fix TestStandbySafeMode.

pritam
authored and
Alex Feinberg
committed
HDFS DataNode: avoid two string concatenation when ClientTraceLog is not enabled

sdong
authored and
Alex Feinberg
committed
Allow Namenode to exclude/deprioritize datanodes which do not heartbeat with the namenode

aaiyer
authored and
Alex Feinberg
committed
Do not update replication queues in removeStoredBlock when replication queues are not initialized.

pritam
authored and
Alex Feinberg
committed
Add retries for connecting zk

tomasz
authored and
Alex Feinberg
committed
BlockDecompressorStream not to return -1 on IOExeption until EOFException

sdong
authored and
Alex Feinberg
committed
Make sure remote job tracker exits the proces

rvadali
authored and
Alex Feinberg
committed
Fix build error

rvadali
authored and
Alex Feinberg
committed
Fix problem with upgrade

tomasz
authored and
Alex Feinberg
committed
Gracefully recovery from runtime exceptions in logSync

avf
authored and
Alex Feinberg
committed
savedLocalDirs should be null, not ""

aching
authored and
Alex Feinberg
committed
Send IP address as part of resource grant

jeanxu
authored and
Alex Feinberg
committed
Improve task launching logic in corona job tracker

jeanxu
authored and
Alex Feinberg
committed
Enhance image/edits config validator for supporting uris

tomasz
authored and
Alex Feinberg
committed
Fix saveVersion for svn checkouts

tomasz
authored and
Alex Feinberg
committed
Revert "Enhance image/edits config validator for supporting uris"

tomasz
authored and
Alex Feinberg
committed
Enhance image/edits config validator for supporting uris

tomasz
authored and
Alex Feinberg
committed
[HDFS Tool] Parse namenode generation stamp and output a fake block for block image viewer

weiyan
authored and
Alex Feinberg
committed
D582382 breaks Unit test

jeanxu
authored and
Alex Feinberg
committed
FsShellService: add skipTrash option to delete

sdong
authored and
Alex Feinberg
committed
Allow hadoop daemons to run as root

pkhemani
authored and
Alex Feinberg
committed
Make sure HttpServer on the namenode shuts down quickly.

pritam
authored and
Alex Feinberg
committed
Checking if file has been uploaded to DistributedCache without fetching whole content

stupaq
authored and
Alex Feinberg
committed
Avoid waiting for ever on stuck locality thread.

rvadali
authored and
Alex Feinberg
committed
Test of HDFS delete() on file being written

stupaq
authored and
Alex Feinberg
committed
Fix assert in NNStorage

tomasz
authored and
Alex Feinberg
committed
Fix bug in FSEditLog

tomasz
authored and
Alex Feinberg
committed
Improve shuffle metrics (more accurate success/failure/serious failure)

aching
authored and
Alex Feinberg
committed
FSShell copy uses wrong file system to check isDirectory()

sdong
authored and
Alex Feinberg
committed
Track the number of java processes running on a node

aching
authored and
Alex Feinberg
committed
Fix race condition in processing revoked grants

rvadali
authored and
Alex Feinberg
committed
Fix bug in FSEditLog

tomasz
authored and
Alex Feinberg
committed
Interrupt lease manager before checking the block count

tomasz
authored and
Alex Feinberg
committed
Add validation of standby/secondary namenode

tomasz
authored and
Alex Feinberg
committed
Add "include" option for build-contrib.xml

tomasz
authored and
Alex Feinberg
committed
Job setup, job cleanup, and task cleanup tasks need their own jvm options

aching
authored and
Alex Feinberg
committed
Task error classification for corona.

rvadali
authored and
Alex Feinberg
committed
Fix dfsclusterhealth

tomasz
authored and
Alex Feinberg
committed
Avoid DNS lookup if IP address is already available.

rvadali
authored and
Alex Feinberg
committed
Extend bug fixes done in rH26579 to AvatarDataNode

sdong
authored and
Alex Feinberg
committed
Instantiate only a single lease checker per namenode in Fastcopy.

pritam
authored and
Alex Feinberg
committed
Enable private actions always in corona jsps

rvadali
authored and
Alex Feinberg
committed
fix build error

rvadali
authored and
Alex Feinberg
committed
Disable TestUlimit - it is failing the nightly build. Need to figure out the problem separately

rvadali
authored and
Alex Feinberg
committed
LocalJobRunner generates task completion events

leizha
authored and
Alex Feinberg
committed
RuntimeMonitor for running JUnit tests

tomasz
authored and
Alex Feinberg
committed
Investigate more timeouts for failover test

tomasz
authored and
Alex Feinberg
committed
Fix file descriptor leak in CachingAvatarZookeeperClient.

pritam
authored and
Alex Feinberg
committed
Make sure LocalityStats.record uses hostname instead of IPaddress

jeanxu
authored and
Alex Feinberg
committed
Enable restoring failed storage by default

tomasz
authored and
Alex Feinberg
committed
Make the port range smaller for avatar tests

tomasz
authored and
Alex Feinberg
committed
Log error for after safemode processing

tomasz
authored and
Alex Feinberg
committed
Make streaming job input key/value serialization configurable

mbautin
authored and
Alex Feinberg
committed
Purge corona system and history directories

jeanxu
authored and
Alex Feinberg
committed
Remove logging for invalidation of blocks when loading edits

tomasz
authored and
Alex Feinberg
committed
Pass digestinputstream to loader instead of DIS in FSImage.loadImage()

tomasz
authored and
Alex Feinberg
committed
OfflineImageDecompressor tool

tomasz
authored and
Alex Feinberg
committed
Introduce logic for max number of failed checkpoints

tomasz
authored and
Alex Feinberg
committed
Disallow to call save namespace for avatarnode-standby

tomasz
authored and
Alex Feinberg
committed
[AUTOCONF] moving autoconf under hadoop-0.20

alushnikov
authored and
Alex Feinberg
committed
Illegal pool names should not be allow in configured pools mode

aching
authored and
Alex Feinberg
committed
Do not shutdown standby when failover does not succeed.

pritam
authored and
Alex Feinberg
committed
Limit class name length for logging messages

tomasz
authored and
Alex Feinberg
committed
Prevent high-memory jobs from starting

alessandro
authored and
Alex Feinberg
committed
[AUTOCONF] setting default values for config

alushnikov
authored and
Alex Feinberg
committed
Log the memory threshold when rejecting the job.

rvadali
authored and
Alex Feinberg
committed
Allow killing succeeded tasks from web UI

rvadali
authored and
Alex Feinberg
committed
Introduce retries for AvatarNode-ZK interaction

tomasz
authored and
Alex Feinberg
committed
Fix testcase TestAvatarStartup

tomasz
authored and
Alex Feinberg
committed
Deletion with hardlinks doesn't work - fix

tomasz
authored and
Alex Feinberg
committed
Enhance the unit tests for hardlinks

liyintang
authored and
Alex Feinberg
committed
Instrument the number of reducers which process no data(Hadoop changes)

adobriyal
authored and
Alex Feinberg
committed
[BKJM-01-ivy] Add BookKeeper dependency to ivy

avf
authored and
Alex Feinberg
committed
Default max task memory = 4096m

rvadali
authored and
Alex Feinberg
committed
Revert "[BKJM-01-ivy] Add BookKeeper dependency to ivy"

tomasz
authored and
Alex Feinberg
committed
Remove the possibility of not enabling directory restoration

tomasz
authored and
Alex Feinberg
committed
Fix ingest bugs

tomasz
authored and
Alex Feinberg
committed
HDFS: fix TestFileAppend4

sdong
authored and
Alex Feinberg
committed
Upgrade ivy to 2.1.0

avf
authored and
Alex Feinberg
committed
Fix NPE in getFileChecksum

hkuang
authored and
Alex Feinberg
committed
[BKJM-01-ivy] Add BookKeeper dependency to ivy (re-submit)

avf
authored and
Alex Feinberg
committed
Avoid InetSocketAddress.getHostName()

rvadali
authored and
Alex Feinberg
committed
Enable buffered IO for log4j

tomasz
authored and
Alex Feinberg
committed
Check the log disk free space

rvadali
authored and
Alex Feinberg
committed
Get the included hosts without copying

rvadali
authored and
Alex Feinberg
committed
Fix the cleanup of jtlogs

rvadali
authored and
Alex Feinberg
committed
Remove getFilesTotal() from FSNamesystem

pritam
authored and
Alex Feinberg
committed
HDFS Inline Checksum

sdong
authored and
Alex Feinberg
committed
Enhance AvatarShell to support an interactive CLI for failover.

pritam
authored and
Alex Feinberg
committed
DFSClient.getFileChecksum() to try other data nodes for connection error

sdong
authored and
Alex Feinberg
committed
Fix TestOverReplicatedBlocks

tomasz
authored and
Alex Feinberg
committed
Fixes for java7 - batch 2

tomasz
authored and
Alex Feinberg
committed
Add syncing when writing to ZK.

tomasz
authored and
Alex Feinberg
committed
Fixes for java7 - batch 1

tomasz
authored and
Alex Feinberg
committed
Introduce timeout for http connection for image download

tomasz
authored and
Alex Feinberg
committed
corona: show the pool name for active jobs

djbw
authored and
Alex Feinberg
committed
Add purging of ckpt images

tomasz
authored and
Alex Feinberg
committed
HDFS Inline Checksum to add version number in file name and eliminate header

sdong
authored and
Alex Feinberg
committed
Clean up checkpoint properly on primary/standby

tomasz
authored and
Alex Feinberg
committed
Fix the "ignoreKey" flag in Hadoop Streaming

mbautin
authored and
Alex Feinberg
committed
Fix NPE in transfer fsimage

tomasz
authored and
Alex Feinberg
committed
Fix TestAvatarContinuousFailover.

pritam
authored and
Alex Feinberg
committed
HDFS: Fix bug of truncateBlock() when no partial chunk

sdong
authored and
Alex Feinberg
committed
Fix a regression caused by removing header from inline checksum

sdong
authored and
Alex Feinberg
committed
Fix several test cases for inline checksum

tomasz
authored and
Alex Feinberg
committed
Updated corona cluster manager freeslot counts to pages that provide details about the free nodes. Information about their cpus, memory, disk space, and network speed are now linked to from cm.jsp

neilc
authored and
Alex Feinberg
committed
Pulled revision 29960 that had improperly formatted log message

neilc
authored and
Alex Feinberg
committed
Provide more details of unallocated resources in Corona

neilc
authored and
Alex Feinberg
committed
HDFS: Fix TestDFSShell after removing header for inlinechecksum files

sdong
authored and
Alex Feinberg
committed
Fix problem with NPE in transfer fs image

tomasz
authored and
Alex Feinberg
committed
Remove ivy-2.0.0 jar, add ivy-2.1.0 jar
Alex Feinberg
committed
Parse "True" and "1" as "true", "False" and "0" as "false"

mbautin
authored and
Alex Feinberg
committed
Simplify standby backup logic

tomasz
authored and
Alex Feinberg
committed
Disable standby backup limits for tests

tomasz
authored and
Alex Feinberg
committed
TestFileStatusExtended failure

tomasz
authored and
Alex Feinberg
committed
Fix Date Time Format

pamelavagata
authored and
Alex Feinberg
committed
add seqNo between Corona cluster manager and CoronaJobTracker

jeanxu
authored and
Alex Feinberg
committed
Another source of mismatched generation stamp replicas

hkuang
authored and
Alex Feinberg
committed
Fix OfferService ZK interaction

tomasz
authored and
Alex Feinberg
committed
Fix bug where datanode couldn't register and continuously sent block reports

pritam
authored and
Alex Feinberg
committed
Inline checksums - fix problem with block report

tomasz
authored and
Alex Feinberg
committed
Make the injections to be the last statements of doCheckpoint

tomasz
authored and
Alex Feinberg
committed
Fix TestAvatarCleanShutdown conflict with host file name

tomasz
authored and
Alex Feinberg
committed
Enable private actions always coronajobdetails.jsp

rvadali
authored and
Alex Feinberg
committed
Fix the free disk space api.

rvadali
authored and
Alex Feinberg
committed
Fixes for java7 - batch 3

tomasz
authored and
Alex Feinberg
committed
Fix broken link for failed tasks generated by corona job tracker

zhguo
authored and
Alex Feinberg
committed
Fix TestAvatarCheckpointing

tomasz
authored and
Alex Feinberg
committed
[DistCp Tos] support the TOS setting in DistCp

dikang
authored and
Alex Feinberg
committed
[HDFS Namenode BugFix] process over replictaed block asynly

weiyan
authored and
Alex Feinberg
committed
Fix reporting of received blocks when they are rejected

tomasz
authored and
Alex Feinberg
committed
Fix more failures in TestAvatarCheckpointing

tomasz
authored and
Alex Feinberg
committed
Online upgrade of task trackers

jeanxu
authored and
Alex Feinberg
committed
Max Map-Tasks for Map-Reduce Jobs

amirr
authored and
Alex Feinberg
committed
Ant rule to generate a single jar with all classes

rvadali
authored and
Alex Feinberg
committed
[Hdfs Fix] Fix the TestNativeIO

dikang
authored and
Alex Feinberg
committed
Fix http://ci-builds.fb.com/job/build_hadoop_warehouse_percommit_trunk/473/

jeanxu
authored and
Alex Feinberg
committed
Backup Corona releases so that Corona job tracker will not be interrupteded by new relases

jeanxu
authored and
Alex Feinberg
committed
No exception if releaseDir is not setup in conf for CoronaReleaseManager

jeanxu
authored and
Alex Feinberg
committed
Make storage directory failures consistent

tomasz
authored and
Alex Feinberg
committed
Fix problem with handling failed journals

tomasz
authored and
Alex Feinberg
committed
Create unit test for CoronaReleaseManager from https://phabricator.fb.com/D602268

jeanxu
authored and
Alex Feinberg
committed
Trace more details (and cleanup)

tylerharter
authored and
Alex Feinberg
committed
Reducer TaskCompletion events are not at all populated for the hive query in the silver cluster

adobriyal
authored and
Alex Feinberg
committed
Speed up offline image viewer

tomasz
authored and
Alex Feinberg
committed
Fail namenode when journal set is empty or editlog stream is null

tomasz
authored and
Alex Feinberg
committed
setTimes doesn't change the modificationTime of a working copy's tag file

jeanxu
authored and
Alex Feinberg
committed
Fix TestAvatarStartup

tomasz
authored and
Alex Feinberg
committed
Add check to avoid NPE in FSNamesystem close.

tomasz
authored and
Alex Feinberg
committed
Revert "Fix TestAvatarStartup"

tomasz
authored and
Alex Feinberg
committed
DFSAdmin#getBlockInfo prints block generation stamp

hkuang
authored and
Alex Feinberg
committed
Disallow non-initial block reports to add mismatched GS replicas into blockMap

hkuang
authored and
Alex Feinberg
committed
Fix TestAvatarStartup - attempt2

tomasz
authored and
Alex Feinberg
committed
More optimizations for offline image viewer

tomasz
authored and
Alex Feinberg
committed
Command line interface for testing expire files logic

rvadali
authored and
Alex Feinberg
committed
Memory leak on standby's

tomasz
authored and
Alex Feinberg
committed
Fix AvatarNode cleaner interval

tomasz
authored and
Alex Feinberg
committed
Remove excessive logging

rvadali
authored and
Alex Feinberg
committed
Enhance metasave for saving excess blocks

tomasz
authored and
Alex Feinberg
committed
Reduce pendingReplicationCheck granularity

hkuang
authored and
Alex Feinberg
committed
Fix TestPersistTxId after Injection handler refactorization.

tomasz
authored and
Alex Feinberg
committed
Fix bin-package for snappy

rvadali
authored and
Alex Feinberg
committed
Enabled job file cleanup always

rvadali
authored and
Alex Feinberg
committed
Fix task error counters

rvadali
authored and
Alex Feinberg
committed
Added TestCompletionEvents

rvadali
authored and
Alex Feinberg
committed
Add getRedirect to ClusterManager

pamelavagata
authored and
Alex Feinberg
committed
Commiting Thrift Generated files

pamelavagata
authored and
Alex Feinberg
committed
HDFS: Fix DFSClient's memory leak on OutputStreams hold by FileChecker

sdong
authored and
Alex Feinberg
committed