alluxio / community Goto Github PK

New Contributor Tasks for Alluxio

community's Introduction

What is Alluxio

Alluxio (formerly known as Tachyon) is a virtual distributed storage system. It bridges the gap between computation frameworks and storage systems, enabling computation applications to connect to numerous storage systems through a common interface. Read more about Alluxio Overview.

The Alluxio project originated from a research project called Tachyon at AMPLab, UC Berkeley, which was the data layer of the Berkeley Data Analytics Stack (BDAS). For more details, please refer to Haoyuan Li's PhD dissertation Alluxio: A Virtual Distributed File System.

Who Uses Alluxio

Alluxio is used in production to manage Petabytes of data in many leading companies, with the largest deployment exceeding 3,000 nodes. You can find more use cases at Powered by Alluxio or visit our first community conference (Data Orchestration Summit) to learn from other community members!

Who Owns and Manages Alluxio Project

Alluxio Open Source Foundation is the owner of Alluxio project. Project operation is done by Alluxio Project Management Committee (PMC). You can checkout more details in its structure and how to join Alluxio PMC here.

Community and Events

Please use the following to reach members of the community:

Alluxio Community Slack Channel: post your questions here if you seek for help for general questions or issues using Alluxio.
Special Interest Groups (SIG) for Alluxio users and developers
Community Events: upcoming online office hours, meetups and webinars
Meetup Groups: Global Online Meetup, Bay Area Meetup, New York Meetup, Beijing Alluxio Meetup, Austin Meetup
Alluxio Twitter; Alluxio Youtube Channel; Alluxio Mailing List

Download Alluxio

Binary download

Prebuilt binaries are available to download at https://www.alluxio.io/download .

Docker

Download and start an Alluxio master and a worker. More details can be found in documentation.

# Create a network for connecting Alluxio containers
$ docker network create alluxio_nw
# Create a volume for storing ufs data
$ docker volume create ufs
# Launch the Alluxio master
$ docker run -d --net=alluxio_nw \
    -p 19999:19999 \
    --name=alluxio-master \
    -v ufs:/opt/alluxio/underFSStorage \
    alluxio/alluxio master
# Launch the Alluxio worker
$ export ALLUXIO_WORKER_RAMDISK_SIZE=1G
$ docker run -d --net=alluxio_nw \
    --shm-size=${ALLUXIO_WORKER_RAMDISK_SIZE} \
    --name=alluxio-worker \
    -v ufs:/opt/alluxio/underFSStorage \
    -e ALLUXIO_JAVA_OPTS="-Dalluxio.worker.ramdisk.size=${ALLUXIO_WORKER_RAMDISK_SIZE} -Dalluxio.master.hostname=alluxio-master" \
    alluxio/alluxio worker

MacOS Homebrew

$ brew install alluxio

Quick Start

Please follow the Guide to Get Started to run a simple example with Alluxio.

Report a Bug

To report bugs, suggest improvements, or create new feature requests, please open a Github Issue. If you are not sure whether you run into bugs or simply have general questions with respect to Alluxio, post your questions on Alluxio Slack channel.

Depend on Alluxio

Alluxio project provides several different client artifacts for external projects to depend on Alluxio client:

Artifact alluxio-shaded-client is recommended generally for a project to use Alluxio client. The jar of this artifact is self-contained (including all dependencies in a shaded form to prevent dependency conflicts), and thus larger than the following two artifacts.
Artifact alluxio-core-client-fs provides Alluxio Java file system API) to access all Alluxio-specific functionalities. This artifact is included in alluxio-shaded-client.
Artifact alluxio-core-client-hdfs provides HDFS-Compatible file system API. This artifact is included in alluxio-shaded-client.

Here are examples to declare the dependencies on alluxio-shaded-client using Maven:

<dependency>
  <groupId>org.alluxio</groupId>
  <artifactId>alluxio-shaded-client</artifactId>
  <version>2.6.0</version>
</dependency>

Contributing

Contributions via GitHub pull requests are gladly accepted from their original author. Along with any pull requests, please state that the contribution is your original work and that you license the work to the project under the project's open source license. Whether or not you state this explicitly, by submitting any copyrighted material via pull request, email, or other means you agree to license the material under the project's open source license and warrant that you have the legal authority to do so. For a more detailed step-by-step guide, please read how to contribute to Alluxio. For new contributor, please take two new contributor tasks.

For advanced feature requests and contributions, Alluxio core team is hosting regular online meetings with community users and developers to iterate the project in two special interest groups:

Alluxio and AI workloads: e.g., running Tensorflow, Pytorch on Alluxio through the POSIX API. Checkout the meeting notes
Alluxio and Presto workloads: e.g., running Presto on Alluxio. Checkout the meeting notes

Subscribe our public calendar to join us.

Useful Links

community's People

Contributors

Stargazers

Watchers

community's Issues

Deprecate Throwables.propagate with RuntimeException in Configuration.java

Guava "Throwables.propagate" is deprecated. see reasons in
https://github.com/google/guava/wiki/Why-we-deprecated-Throwables.propagate

This ticket modifies https://github.com/Alluxio/alluxio/blob/master/core/common/src/main/java/alluxio/Configuration.java#L418 to replace

throw Throwables.propagate(e);

Throwables.propagate(e);

with

throw new RuntimeException(e);

Remember also to remove the corresponding import if Throwables becomes unused

import com.google.common.base.Throwables;

Deprecate Throwables.propagate with RuntimeException in UnderFileSystemUtils.java

Guava "Throwables.propagate" is deprecated. see reasons in
https://github.com/google/guava/wiki/Why-we-deprecated-Throwables.propagate

This ticket modifies https://github.com/Alluxio/alluxio/blob/master/core/common/src/main/java/alluxio/util/UnderFileSystemUtils.java#L82 to replace

throw Throwables.propagate(e);

Throwables.propagate(e);

with

throw new RuntimeException(e);

Remember also to remove the corresponding import if Throwables becomes unused

import com.google.common.base.Throwables;

Rename LoadMetadataTest.java to be LoadMetadataIntegrationTest.java

Rename LoadMetadataTest.java to be LoadMetadataIntegrationTest.java

Class name also needs to be changed correspondingly.

Improve javadoc for MetricsConfig#removeInstancePrefix

Instead of "This method removes the instance prefix ...", the javadoc should say "Removes the instance prefix ..."

Fix spelling in JournalReaderOptions

In /core/server/common/src/main/java/alluxio/master/journal/options/JournalReaderOptions.java, for the method: setPrimary(boolean primary), fix the spelling in the comments:

jorunal -> journal

Improve Worker Data Folder property key description

In PropertyKey.java

Update

public static final PropertyKey WORKER_DATA_HOSTNAME =
      new Builder(Name.WORKER_DATA_HOSTNAME)
          .setDescription("The hostname of Alluxio worker data service").build();

public static final PropertyKey WORKER_DATA_HOSTNAME =
      new Builder(Name.WORKER_DATA_HOSTNAME)
          .setDescription("The hostname of Alluxio worker data service.").build();

Deprecate Throwables.propagate with RuntimeException in OutStreamOptions.java

Guava "Throwables.propagate" is deprecated. see reasons in
https://github.com/google/guava/wiki/Why-we-deprecated-Throwables.propagate

This ticket modifies https://github.com/Alluxio/alluxio/blob/master/core/client/fs/src/main/java/alluxio/client/file/options/OutStreamOptions.java#L68 to replace

throw Throwables.propagate(e);

Throwables.propagate(e);

with

throw new RuntimeException(e);

Remember also to remove the corresponding import if Throwables becomes unused

import com.google.common.base.Throwables;

Fix formatting after Nullable in DefaultFileSystemMaster

In core/server/master/src/main/java/alluxio/master/file/DefaultFileSystemMaster.java, we have the signature

private FileSystemMasterAuditContext createAuditContext(String command, AlluxioURI srcPath,
    @Nullable  AlluxioURI dstPath, @Nullable  Inode srcInode) throws AccessControlException {

There are two spaces after each @Nullable annotation, but there should be only one

Remove blank line in DataByteBuffer

The constructor goes

  /**
   *
   * @param buffer The ByteBuffer representing the data
   * @param length The length of the ByteBuffer
   */

We should delete the blank line before the parameters

Improve javadoc for DataBuffer#release

Fix the tense by changing "Release the underlying buffer" to "Releases the underlying buffer"

Improve Naming in FileSystemAclIntegrationTest

In the test directoryPermissionForUfs, rename the directory path from /root/dir/ to /root/directoryPermissionForUfsDir

Update Web-Interface.md to reflect the latest change in UI

Alluxio web UI is recently updated to show in-Alluxio data instead of in-memory data. Please update /docs/en/Web-Interface.md and replace Files currently in the memory tier with Files currently in Alluxio

Rename ReadHandlerTest to AbstractReadHandlerTest

Rename ReadHandlerTest.java to AbstractReadHandlerTest

Note that, the class name also needs to be changed correspondingly.

Mark alluxio.client.file.FileInStream#mStatus final

Improve core/client/fs/src/main/java/alluxio/client/file/FileInStream.java as mStatus can be marked as final.

Change

  protected URIStatus mStatus;

  protected final URIStatus mStatus;

Rename ConcurrentFileSystemMasterCreateTest.java to ConcurrentCreateIntegrationTest.java

Rename ConcurrentFileSystemMasterCreateTest.java to ConcurrentCreateIntegrationTest.java

Note that, the class name also needs to be changed correspondingly.

Deprecate Throwables.propagate with RuntimeException in InStreamOptions.java

Guava "Throwables.propagate" is deprecated. see reasons in
https://github.com/google/guava/wiki/Why-we-deprecated-Throwables.propagate

This ticket modifies https://github.com/Alluxio/alluxio/blob/master/core/client/fs/src/main/java/alluxio/client/file/options/InStreamOptions.java#L63 to replace

throw Throwables.propagate(e);

Throwables.propagate(e);

with

throw new RuntimeException(e);

Remember also to remove the corresponding import if Throwables becomes unused

import com.google.common.base.Throwables;

Improve Naming for FileSystemAclIntegrationTest

In the test swiftGetPermission, rename the file /objectfileA to /swiftGetPermissionFile

Update comment in JournalShutdownIntegrationTest

In tests/src/test/java/alluxio/master/JournalShutdownIntegrationTest.java, we have the sentence "Test both the single master (alluxio) and multi masters (alluxio-ft)." alluxio-ft has been deprecated in version 1.6.0, so we should update the sentence to just "Test both the single master and multi masters."

Rename WriteHandlerTest to AbstractWriteHandlerTest

Rename WriteHandlerTest.java to AbstractWriteHandlerTest

Note that, the class name also needs to be changed correspondingly.

Update preconditions in AbstractUfsManager.java

In /core/server/common/src/main/java/alluxio/underfs/AbstractUfsManager.java, in method addMount(long mountId, final AlluxioURI ufsUri, final UnderFileSystemConfiguration ufsConf), change the preconditions:

Preconditions.checkArgument(ufsUri != null, "ufsUri");
Preconditions.checkArgument(ufsConf != null, "ufsConf");

Preconditions.checkNotNull(ufsUri, "ufsUri");
Preconditions.checkNotNull(ufsConf, "ufsConf");

Mark alluxio.client.file.FileOutStream#mPreviousBlockOutStreams final

Improve core/client/fs/src/main/java/alluxio/client/file/FileOutStream.java as mPreviousBlockOutStreams can be marked as final.

Change

  private List<BlockOutStream> mPreviousBlockOutStreams;

  private final List<BlockOutStream> mPreviousBlockOutStreams;

Deprecate Throwables.propagate with RuntimeException in ManuallyScheduleHeartbeat.java

Guava "Throwables.propagate" is deprecated. see reasons in
https://github.com/google/guava/wiki/Why-we-deprecated-Throwables.propagate

This ticket modifies https://github.com/Alluxio/alluxio/blob/master/core/common/src/test/java/alluxio/heartbeat/ManuallyScheduleHeartbeat.java#L65 to replace

throw Throwables.propagate(e);

Throwables.propagate(e);

with

throw new RuntimeException(e);

Remember also to remove the corresponding import if Throwables becomes unused

import com.google.common.base.Throwables;

Deprecate Throwables.propagate with RuntimeException in CommandUtils.java

Guava "Throwables.propagate" is deprecated. see reasons in
https://github.com/google/guava/wiki/Why-we-deprecated-Throwables.propagate

This ticket modifies https://github.com/Alluxio/alluxio/blob/cf47f8ba1761490340c9684aee24377a2cd492c2/core/common/src/main/java/alluxio/cli/CommandUtils.java#L53 to replace

throw Throwables.propagate(e);

Throwables.propagate(e);

with

throw new RuntimeException(e);

Remember also to remove the corresponding import if Throwables becomes unused

import com.google.common.base.Throwables;

Fix man page description of setTtl

The man page in shell/src/main/resources/fs.man has the sentence "Set the TTL (time to live) in milliseconds to a file". This should say "Set the TTL (time to live) in milliseconds for a file"

Improve Naming for FileSystemAclIntegrationTest

In the test gcsGetPermission, rename the file /objectFileA to /gcsGetPermissionFile

Improve javadoc for MasterUfsManager

Update

/**
 * A class manages the UFS used by different services.
 */

/**
 * A class that manages the UFS for master servers.
 */

Deprecate Throwables.propagate with RuntimeException in EmbeddedChannels.java

Guava "Throwables.propagate" is deprecated. see reasons in
https://github.com/google/guava/wiki/Why-we-deprecated-Throwables.propagate

This ticket modifies https://github.com/Alluxio/alluxio/blob/master/core/common/src/test/java/alluxio/EmbeddedChannels.java#L90 to replace

throw Throwables.propagate(e);

Throwables.propagate(e);

with

throw new RuntimeException(e);

Remember also to remove the corresponding import if Throwables becomes unused

import com.google.common.base.Throwables;

Rename ConcurrentFileSystemMasterDeleteTest.java to ConcurrentDeleteIntegrationTest.java

Rename ConcurrentFileSystemMasterDeleteTest.java to ConcurrentDeleteIntegrationTest.java

Note that, the class name also needs to be changed correspondingly.

Rename ConcurrentFileSystemMasterRenameTest.java to ConcurrentRenameIntegrationTest.java

Rename ConcurrentFileSystemMasterRenameTest.java to ConcurrentRenameIntegrationTest.java

Note that, the class name also needs to be changed correspondingly.

Improve javadoc in MetricsSystem#stripInstanceAndHost

Update the javadoc from

Util function to remove get the metrics name without instance and host.
to
Removes the instance and host from the given metric name, returning the result.

Rename ConcurrentFileSystemMasterSetTtlTest.java to ConcurrentSetTtlIntegrationTest.java

Rename ConcurrentFileSystemMasterSetTtlTest.java to ConcurrentSetTtlIntegrationTest.java

Note that, the class name also needs to be changed correspondingly.

Improve Swift Auth Method description in PropertyKey

In PropertyKey.java, update

  public static final PropertyKey SWIFT_AUTH_METHOD_KEY = new Builder(Name.SWIFT_AUTH_METHOD_KEY)
      .setDescription("Choice of authenitcation method: "
          + "[tempauth (default), swiftauth, keystone, keystonev3].")

  public static final PropertyKey SWIFT_AUTH_METHOD_KEY = new Builder(Name.SWIFT_AUTH_METHOD_KEY)
      .setDescription("Choice of authentication method: "
          + "[tempauth (default), swiftauth, keystone, keystonev3].")

Fix description for setTtl command

The description in docs/_data/table/en/operation-command.yml says "Set the TTL (time to live) in milliseconds to a file". This should be "Set the TTL (time to live) in milliseconds for a file".

Remove unused variable in FileSystemAclIntegrationTest

The variable BLOCK_SIZE is no longer used.

Update class comments for DefaultBlockMaster

In /core/server/master/src/main/java/alluxio/master/block/DefaultBlockMaster.java for the class comments, update:

This master manages the...

This block master manages the...

Deprecate Throwables.propagate with RuntimeException in CommonTestUtils.java

Guava "Throwables.propagate" is deprecated. see reasons in
https://github.com/google/guava/wiki/Why-we-deprecated-Throwables.propagate

This ticket modifies https://github.com/Alluxio/alluxio/blob/master/core/common/src/test/java/alluxio/CommonTestUtils.java#L109
https://github.com/Alluxio/alluxio/blob/master/core/common/src/test/java/alluxio/CommonTestUtils.java#L132
to replace

throw Throwables.propagate(e);

Throwables.propagate(e);

with

throw new RuntimeException(e);

Remember also to remove the corresponding import if Throwables becomes unused

import com.google.common.base.Throwables;

Fix comment in InodeTree

In core/server/master/src/main/java/alluxio/master/file/meta/InodeTree.java the javadoc says
"Represents the tree of Inode's". This should be "Inodes", not "Inode's".

Deprecate Throwables.propagate with RuntimeException in OpenFileOptions.java

Guava "Throwables.propagate" is deprecated. see reasons in
https://github.com/google/guava/wiki/Why-we-deprecated-Throwables.propagate

This ticket modifies https://github.com/Alluxio/alluxio/blob/master/core/client/fs/src/main/java/alluxio/client/file/options/OpenFileOptions.java#L63 to replace

throw Throwables.propagate(e);

Throwables.propagate(e);

with

throw new RuntimeException(e);

Remember also to remove the corresponding import if Throwables becomes unused

import com.google.common.base.Throwables;

Remove blank line in DataByteArrayChannel constructor

The constructor javadoc goes

  /**
   *
   * @param byteArray the array representing the data
   * @param offset the offset into the byteArray
   * @param length the length of the data
   */

We should delete the blank line before the parameters

Improve Master Format File Prefix description in PropertyKey

In PropertyKey.java, update

  public static final PropertyKey MASTER_FORMAT_FILE_PREFIX =
      new Builder(Name.MASTER_FORMAT_FILE_PREFIX)
          .setDefaultValue("_format_")
          .setDescription("The file prefix of the file generated in the journal directory "
              + "when the journal is formatted. The master will search for a file with this "
              + "prefix when determining of the journal was once formatted.")

  public static final PropertyKey MASTER_FORMAT_FILE_PREFIX =
      new Builder(Name.MASTER_FORMAT_FILE_PREFIX)
          .setDefaultValue("_format_")
          .setDescription("The file prefix of the file generated in the journal directory "
              + "when the journal is formatted. The master will search for a file with this "
              + "prefix when determining if the journal is formatted.")

This ticket modifies
https://github.com/Alluxio/alluxio/blob/master/core/common/src/main/java/alluxio/util/network/NetworkAddressUtils.java#L348
https://github.com/Alluxio/alluxio/blob/master/core/common/src/main/java/alluxio/util/network/NetworkAddressUtils.java#L421
https://github.com/Alluxio/alluxio/blob/master/core/common/src/main/java/alluxio/util/network/NetworkAddressUtils.java#L550
to replace

throw Throwables.propagate(e);

Throwables.propagate(e);

with

throw new RuntimeException(e);

Remember also to remove the corresponding import if Throwables becomes unused

import com.google.common.base.Throwables;

Improve HDFS Prefixes description in PropertyKey

In PropertyKey.java, update

  public static final PropertyKey UNDERFS_HDFS_PREFIXES =
      new Builder(Name.UNDERFS_HDFS_PREFIXES)
          .setDefaultValue("hdfs://,glusterfs:///,maprfs:///")
          .setDescription("Optionally, specify which prefixes should run through the Apache "
              + "Hadoop implementation of UnderFileSystem. The delimiter is any whitespace "
              + "and/or ','.")

  public static final PropertyKey UNDERFS_HDFS_PREFIXES =
      new Builder(Name.UNDERFS_HDFS_PREFIXES)
          .setDefaultValue("hdfs://,glusterfs:///,maprfs:///")
          .setDescription("Optionally, specify which prefixes should run through the HDFS "
              + "implementation of UnderFileSystem. The delimiter is any whitespace "
              + "and/or ','.")

Improve description of HDFS configuration in PropertyKey

In PropertyKey.java, update

  public static final PropertyKey UNDERFS_HDFS_CONFIGURATION =
      new Builder(Name.UNDERFS_HDFS_CONFIGURATION)
          .setDefaultValue(String.format(
              "${%s}/core-site.xml:${%s}/hdfs-site.xml", Name.CONF_DIR, Name.CONF_DIR))
          .setDescription("Location of the hdfs configuration file.")
          .build();

  public static final PropertyKey UNDERFS_HDFS_CONFIGURATION =
      new Builder(Name.UNDERFS_HDFS_CONFIGURATION)
          .setDefaultValue(String.format(
              "${%s}/core-site.xml:${%s}/hdfs-site.xml", Name.CONF_DIR, Name.CONF_DIR))
          .setDescription("Location of the HDFS configuration file.")
          .build();

Remove blank line in DataFileChannel

The constructor goes

  /**
   *
   * @param file The file
   * @param offset The offset into the FileChannel
   * @param length The length of the data to read
   */

We should delete the blank line before the parameters

Deprecate Throwables.propagate with RuntimeException in CreateFileOptions.java

Guava "Throwables.propagate" is deprecated. see reasons in
https://github.com/google/guava/wiki/Why-we-deprecated-Throwables.propagate

This ticket modifies https://github.com/Alluxio/alluxio/blob/master/core/client/fs/src/main/java/alluxio/client/file/options/CreateFileOptions.java#L67 to replace

throw Throwables.propagate(e);

Throwables.propagate(e);

with

throw new RuntimeException(e);

Remember also to remove the corresponding import if Throwables becomes unused

import com.google.common.base.Throwables;

Update comments in PrimarySelectorClient.java

In /core/server/common/src/main/java/alluxio/master/PrimarySelectorClient.java, in method getNewCuratorClient(), update the comments from:

// state, explicitly close the "old" client recreate a new one.

// state, explicitly close the "old" client and recreate a new one.

alluxio / community Goto Github PK

community's Introduction

What is Alluxio

Who Uses Alluxio

Who Owns and Manages Alluxio Project

Community and Events

Download Alluxio

Binary download

Docker

MacOS Homebrew

Quick Start

Report a Bug

Depend on Alluxio

Contributing

Useful Links

community's People

Contributors

Stargazers

Watchers

Forkers

community's Issues

Recommend Projects

Recommend Topics

Recommend Org