Code Monkey home page Code Monkey logo

imapnio's Introduction

imapnio

A Java library that supports NIO (Non-blocking I/O) based IMAP clients.

The IMAP NIO client provides a framework to access an IMAP message store using None-blocking I/O mechanism. This design scales well for thousands of connections per machine and reduces contention when using a large number of threads and CPUs.

Table of Contents

Background

The traditional accessing IMAP message store uses JavaMail API, which requires a blocking I/O. In this case, threads are blocked when performing I/O with the other end. This project was developed to relieve the waiting thread to perform other tasks, and it's design efficiently improves thread utilization to maximize hardware throughput and capacity.

Some of the more distinguishing features of this library are:

  • Highly customizable thread model and server/client idle max limit.
  • Leverages the well-established framework Netty
  • Future-based design enables a clean separation of IMAP client threads pool versus consumers threads pool.
  • IMAPv4, RFC3501 support.
  • ID command, RFC2971 support.
  • IDLE command, RFC2177 support
  • MOVE command, RFC6851 support
  • UNSELECT command, RFC3691 support

This project is ideal for applications that have a high requirement to optimize thread utilization and improve overall resource capacity. Specifically, this is best for situations where users perform a very high level of sessions and communication with the IMAP server.

Install

This is a Java library. After downloading it, compile it using mvn clean install

Then, update your project's pom.xml file dependencies, as follows:

  <dependency>
      <groupId>com.yahoo.imapnio</groupId>
      <artifactId>imapnio.core</artifactId>
      <version>5.0.15</version>
  </dependency>

Finally, import the relevant classes and use this library according to the usage section below.

  • For contributors run deploy to do a push to nexus servers
	$ mvn clean deploy -Dgpg.passphrase=[pathPhrase]

Usage

The following code examples demonstrate basic functionality relate to connecting to and communicating with IMAP servers.

Create a client

  // Create a ImapAsyncClient instance with number of threads to handle the server requests
  final int numOfThreads = 5;
  final ImapAsyncClient imapClient = new ImapAsyncClient(numOfThreads);

Establish a session with an IMAP server

  // Create a new session via the ImapAsyncClient instance created above and connect to that server.  For the illustration purpose, 
  // "imaps://imap.server.com:993" is used
  final URI serverUri = new URI("imaps://imap.server.com:993");
  final ImapAsyncSessionConfig config = new ImapAsyncSessionConfig();
  config.setConnectionTimeoutMillis(5000);
  config.setReadTimeoutMillis(6000);
  final List<String> sniNames = null;

  final InetSocketAddress localAddress = null;
  final Future<ImapAsyncCreateSessionResponse> future = imapClient.createSession(serverUri, config, localAddress, sniNames, DebugMode.DEBUG_OFF);
  
  //this version is a future-based nio client.  Check whether future is done by following code.
  if (future.isDone()) {
	System.out.println("Future is done.");
  }

Execute the IMAP command to IMAP server

Following codes uses a Capability command as an example.

  // Executes the capability command
  final Future<ImapAsyncResponse> capaCmdFuture = session.execute(new CapaCommand());

Handle the response from IMAP server

If the future of the executed command is done, obtain the response. Following example shows how to read ImapAsyncResponse which wraps the content sent from the server.

  if (capaCmdFuture.isDone()) {
	System.out.println("Capability command is done.");
	final ImapAsyncResponse resp = capaCmdFuture.get(5, TimeUnit.MILLISECONDS);
	final ImapResponseMapper mapper = new ImapResponseMapper();
	final Capability capa = mapper.readValue(resp.getResponseLines().toArray(new IMAPResponse[0]), Capability.class);
	final List<String> values = capa.getCapability("AUTH");
  }

Release

This release, 2.0.x, is a major release. Changes are:

  • It supports non-blocking IO functionality through providing callers with java.util.concurrent.Future object.
  • Listener-based non-blocking IO capability will not be supported in this release.

This release, 5.0.x, is a major release. Changes are:

  • It supports request and response bytes counting per command

Contribute

Please refer to the contributing.md for information about how to get involved. We welcome issues, questions, and pull requests. Pull Requests are welcome.

Maintainers

Fan Su : [email protected] Vikram Nagulakonda : [email protected]

License

This project is licensed under the terms of the Apache 2.0 open source license. Please refer to LICENSE for the full terms.

imapnio's People

Contributors

alex-yxw avatar chibenwa avatar dependabot[bot] avatar dudeakshay avatar fansu avatar jpowang avatar jui8wang avatar kevinliuyifeng avatar krdev avatar lafa avatar nhant01 avatar nituyahoo avatar nvsnvikram avatar remk avatar retlawrose avatar squintview avatar sundarms avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

imapnio's Issues

How do I read the messages from the IMAP server.

Hi, I'm using the imapnio client.

May I know how do I read the messages after executing the IDLE command ?
Here's the code snippet I'm using:

public void testGmailPlainLoginWithIdle() throws Exception {
        final String gmailServer = "imaps://imap.gmail.com:993";
        final IMAPSession session = theClient.createSession(new URI(gmailServer), new Properties(),
                new TestConnectionListener("testGmailPlainLoginWithIdle"), logManager);
        session.connect();
        Thread.sleep(500);

        IMAPCommandListener listenerToSendIdle = new IMAPCommandListener() {
            @Override
            public void onResponse(IMAPSession session, String tag, List<IMAPResponse> responses) {

                for (IMAPResponse r : responses) {
                    if (r.getTag() != null) {
                        Assert.assertTrue(r.isOK());
                        try {
                            session.executeIdleCommand("t1-idle", new TestCommandListener("testGmailPlainLoginWithIdle"));
                            
                        } catch (SecurityException e) {
                            // TODO Auto-generated catch block
                            e.printStackTrace();
                        } catch (IMAPSessionException e) {
                            // TODO Auto-generated catch block
                            e.printStackTrace();
                        }
                    } else {
                        log.info("testGmailPlainLoginWithIdle " + r, null);
                    }
                }
            }

            @Override
            public void onMessage(IMAPSession session, IMAPResponse response) {
                log.info("onMessage " + response, null);
            }
        };

        IMAPCommandListener listenerToSendSelect = new IMAPCommandListener() {

            @Override
            public void onResponse(IMAPSession session, String tag, List<IMAPResponse> responses) {
                try {
                    session.executeSelectCommand("t01-sel", "Inbox", listenerToSendIdle);
                } catch (IMAPSessionException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
            }

            @Override
            public void onMessage(IMAPSession session, IMAPResponse response) {
                log.info("onMessage " + response, null);
            }
        };

        session.executeLoginCommand("t1", "[email protected]", "12343", listenerToSendSelect);
        Thread.sleep(20000);


        
    }

 class TestCommandListener implements IMAPCommandListener {

        final String logPrefix;

        TestCommandListener(String prefix) {
            logPrefix = prefix;
        }
        @Override
        public void onResponse(IMAPSession session, String tag, List<IMAPResponse> responses) {
            System.out.println("onResponse.. testcmdlistener");
        	for (final IMAPResponse r : responses) {
                log.info(logPrefix + " " + r, null);
                System.out.println(r.readString());
            }
        }

        @Override
        public void onMessage(IMAPSession session, IMAPResponse response) {
        	System.out.println("on Message test cmd listener");
        	log.info(logPrefix + "  " + response, null);
            
        }

    }

I get the notification in the "onMessage()" function of TestCommandListener, But I'm unable to read the email. Can you suggest a way to read the email upon notification ?

How many threads in IMAPClient is needed per mailbox ?

Hi,

I'm using imapnio to poll the mailboxes for incoming mails.

The IMAPClient class takes the number of threads as a parameter.
I want to go to IDLE state on all the mailboxes and get notified when the mailbox receives a mail.

if I create a client with 1 thread: i.e

IMAPClient client = new IMAPClient(1);
how many mailboxes/accounts can it handle ?
I want to go to IDLE state for each account.

Thanks for your help.

IMAPConnectionListener onDisconnect() is not called after timeout

Hi,
I'm using the imapnio api.

Here's the code that creates a session:

properties.setProperty(IMAPSession.CONFIG_CONNECTION_TIMEOUT_KEY, "300000"); //5 mins

properties.setProperty(IMAPSession.CONFIG_IMAP_TIMEOUT_KEY, 180000); //3 mins

session = imapClient.createSession(new URI(serverUrl), properties, new IMAPConnectionListenerImpl(imapClient), logManager);

the IMAPConnenctionListenerImpl implements com.lafaspot.imapnio.listener.IMAPConnectionListener interface.

The onDisconnect() method in that interface is not being called due to timeout.

Can you please help me fix it ?

Performance: imapnio spend most of its time filling stacktraces

Description

An analysis of imapnio with a profiler showed most of its time is spent filling stack traces due to integer parsing.

Screenshot from 2022-04-13 09-18-51

The underlying IMAPResponse from javax.mail does an attempt to parse untagged responses under the format * NUMBER KEYWORD exemple * 36 EXISTS.

However some commands like LIST

Expected Behavior

Avoid filling this stacktrace.

Actual Behavior

Fills the stacktrace and spends a lot of time doing so (40% of CPU time dedicated to imapnio)

Possible Fix

It looks like overriding this behaviour in IMAPResponse not to be feasible (init method doing such a check is private)

Context

I am using imapnio to write an IMAP benchmarking tool: https://github.com/linagora/gatling-imap/ . As such, in order not to impact performance measurement low latencies are paramount on the injection side.

Netty Epoll Bug with lafaspot imapnio client

I'm using Netty 4.1.29.Final.
Java Version 1.8.0_181-b13
Amazon AMI Linux 4.9.119-44.140.amzn1.x86_64 (2017.09)

I' using lafaspot imapnio client which uses netty. [(https://github.com/lafaspot/imapnio)]
I'm running my Java App in Tomcat 8.5.29.

I get the following exception while shutting down the tomcat server.
The NIO threads are not being killed, causing a memory leak.

20-Sep-2018 13:07:36.516 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [IMAPNIO-THREAD-POOL-CUSTOM-1-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:737)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:392)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)

20-Sep-2018 13:07:36.517 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [IMAPNIO-THREAD-POOL-CUSTOM-3-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:737)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:392)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)

This is the code I'm using to create and shutdown threads:

public class IMAPClient {

/** instance id used for debug. */
private final String instanceId = Integer.toString(new Random(System.nanoTime()).nextInt());

/** counter for session. */
private AtomicInteger sessionCounter = new AtomicInteger(1);

/** The netty bootstrap. */
private final Bootstrap bootstrap;

/** Event loop group that will serve all channels for IMAP client. */
private final EventLoopGroup group;

/**
 * Constructs a NIO based IMAP client.
 *
 * @param threads number of threads to be used by IMAP client
 */
public IMAPClient(final int threads) {
    this.bootstrap = new Bootstrap();
    this.group = new NioEventLoopGroup(threads, new DefaultThreadFactory("IMAPNIO-THREAD-POOL-CUSTOM"));
    bootstrap.channel(NioSocketChannel.class);
    bootstrap.group(group);
}


/**
 * Close all of the sessions within a client, and shutdown the event group.
 */
public void shutdown() {
	Future<?> future = this.group.shutdownGracefully();
	try {
		
		if(future != null) {
			future.await();
		}
		boolean isTerminated = this.group.awaitTermination(300, TimeUnit.SECONDS);
		
	} catch (InterruptedException e) {
		
	}
   }
}

I tried the workaround mentioned here https://netty.io/news/2012/09/13/4-0-0-alpha4.html

I added the -Dio.netty.epollBugWorkaround=true to JAVA_OPTS. But still I get this error.

Can you please tell me how to fix this error ?

Remove logback core from main dependencies

Description

java.lang.NoSuchMethodError: 'java.lang.String ch.qos.logback.core.util.EnvUtil.logbackVersion()'

	at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:81)
	at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:77)
	at ch.qos.logback.classic.spi.LogbackServiceProvider.initializeLoggerContext(LogbackServiceProvider.java:50)
	at ch.qos.logback.classic.spi.LogbackServiceProvider.initialize(LogbackServiceProvider.java:41)
	at org.slf4j.LoggerFactory.bind(LoggerFactory.java:183)
	at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:170)
	at org.slf4j.LoggerFactory.getProvider(LoggerFactory.java:455)
	at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:441)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:390)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:416)
	at org.apache.james.mock.smtp.server.testing.MockSmtpServerExtension$DockerMockSmtp.<clinit>(MockSmtpServerExtension.java:43)
	at org.apache.james.mock.smtp.server.testing.MockSmtpServerExtension.<init>(MockSmtpServerExtension.java:77)
	at org.apache.james.mailets.RemoteDeliveryForwardIntegrationTest.<clinit>(RemoteDeliveryForwardIntegrationTest.java:76)

Clashes on versions of logback core.

IMO there's no rationals for forcing this: SLF4J logging facade is plenty enough...

Expected Behavior

Remove logback-core + logback-classic dependencies, keep them as testing dependencies.

Actual Behavior

Included and not compatible with those I have on James side.

I need to do exclusions: that's ugly...

        <dependency>
            <groupId>com.yahoo.smtpnio</groupId>
            <artifactId>imapnio.core</artifactId>
            <version>1.1.0</version>
            <exclusions>
                <exclusion>
                    <groupId>ch.qos.logback</groupId>
                    <artifactId>logback-classic</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>ch.qos.logback</groupId>
                    <artifactId>logback-core</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

Possible Fix

Remove logback-core + logback-classic dependencies, keep them as testing dependencies.

ImapAsyncSession: allow to know if the session is open

Description

I would like to assert the state of a session before submitting requests to it.

Expected Behavior

I would like isChannelClosed to be public.

Actual Behavior

Today it is package private.

Possible Fix

Make it publicly visible

Context

IMAP provisionning script

How find if loginCommand failed due to invalid credentials ?

Hi,

I'm using your imap client.

I would like to know how to check if the login command failed due to invalid credentials.
Currently the callbacks are called irrespective of whether the credentials are valid or invalid.

```
public void testGmailPlainLoginWithIdle(String userName, String password) throws Exception {
        final String gmailServer = "imaps://imap.gmail.com:993";
        final IMAPSession session = theClient.createSession(new URI(gmailServer), new Properties(),
                new IMAPConnectionListenerImpl("testGmailPlainLoginWithIdle"), logManager);
        session.connect();
        Thread.sleep(500);

        IMAPCommandListener listenerToSendIdle = new IMAPCommandListener() {
            @Override
            public void onResponse(IMAPSession session, String tag, List<IMAPResponse> responses) {

                for (IMAPResponse r : responses) {
                    if (r.getTag() != null) {
                       // Assert.assertTrue(r.isOK());
                        try {
                            session.executeIdleCommand("t1-idle"+userName, new IMAPCommandListenerImpl(userName,password));
                        } catch (SecurityException e) {
                            // TODO Auto-generated catch block
                            e.printStackTrace();
                        } catch (IMAPSessionException e) {
                            // TODO Auto-generated catch block
                            e.printStackTrace();
                        }
                    }
                }
            }

            @Override
            public void onMessage(IMAPSession session, IMAPResponse response) {
                System.out.println("LISTENER TO SEND IDLE ON MESSAGE");
            }
        };

        IMAPCommandListener listenerToSendSelect = new IMAPCommandListener() {

            @Override
            public void onResponse(IMAPSession session, String tag, List<IMAPResponse> responses) {
            	
            
                try {
                    session.executeSelectCommand("t01-"+userName, "Inbox", listenerToSendIdle);
                } catch (IMAPSessionException e) {
                    e.printStackTrace();
                }
            }

            @Override
            public void onMessage(IMAPSession session, IMAPResponse response) {
                System.out.println("LISTENER TO SEND SELECT ON MESSAGE");
            }
        };
        
        try{
        IMAPChannelFuture future = session.executeLoginCommand("t1-"+userName, userName, password, listenerToSendSelect);
        }catch(IMAPSessionException e) {
        	e.printStackTrace();
        }

    }

 How do I check whether the function executeLoginCommand failed or Succeeded.
 How do I check if executeSelect command failed or succeeded?
 I tried using future.isSuccess() command. But it returns false for both success and failure case.
  Any answers ?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.