levimcclenny / sa-pinns Goto Github PK

Implementation of the paper "Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism" [AAAI-MLPS 2021]

Home Page: https://arxiv.org/abs/2009.04544

Python 100.00%

adaptive-pinns neural-network pdes stiff-pdes scientific-computing sciml physics-informed-learning pinns deep-learning neural-networks

sa-pinns's Introduction

Hi there 👋

I'm currently a PhD Candidate in the Dept. of Electrical Engineering at Texas A&M in College Station. Additionally I'm working with the Army Research Lab on projects of interest to the DoD.

Research Interests

Deep Learning - specifically Physics-Informed Machine Learning or Scientific Machine Learning. I'm fascinated by the intersection of Deep Learning and Physics. Recently I have been working on algorithmic methodologies to improve training of PINNS, as well as an open source software suite for multi-GPU training of large problem domains for PINNS
Materials Informatics - A lot of my research is centered around materials applications. This is the main application domain of TensorDiffEq as it stands, although it is applicable to any field of science.

Other Interests

I'm interested in a swath of areas of computer science that aren't directly related to my research, notable examples are:

Distributed computing - I love working in distributed environments. The Army Research Lab has granted me an Nvidia DGX Station for my research, which is a platform I use heavily to experiment and build in distributed environments
Compilers/IRs - I just find the intersection of hardware and software fascinating. Recently I have been digging into LLVM, MLIR, as well as implementation of assembly in various architectures to include MIPS, x86, and ARM. I'm mostly a high-level programming language guy (Python, R, etc) but I love learning more about the hardware interface, as well as how to take deep-learning implementations to the edge on 'micro' computing devices. Oftentimes these implementations must be executed eithout operating system oversight, and require creative programming
Computer Vision - early in my PhD I worked heavily in computer vision with applications in materials informatics, and learned a lot about CV.
Writing - Recently, I have been contributing to d2l.ai with this knowledge of CV, Tensorflow, Distributed Computing, etc. The d2l.ai project is an open-source general purpose deep learning textbook covering topics from MLPs to attention mechanisms and online video processing. The textbook is being used at 140+ universities worldwide and is endorsed by high-profiles individuals to include Jensen Huang, CEO of Nvidia. Specifically, I have been contributing code snippets to the Tensorflow implementation of the concepts in the text, as well as contributing technical content on differences between code implementation in Tensorflow, Pytorch, and MXNet.
Reviewing - I have taken part in many technical reviews of Manning Publishing books, to include texts on distributed computing and tensorflow, amongst others. Many are in early-stage writing, and technical reviews are anonymous, so I forgo listing specific texts for now.

sa-pinns's People

Contributors

Stargazers

Watchers

sa-pinns's Issues

Bug (Error) Report

Hi @levimcclenny

A very large weight factor (100) is used in the simulations:

i.e. u_weights = tf.Variable(100*tf.random.uniform([N0, 1]))

However, If we set the weight factor to 1.0 (default value)

This proposed Self-Adaptive PINN will NOT work any more, see the output figure from AC example with the following modification:
u_weights = tf.Variable(1*tf.random.uniform([N0, 1]))

at t = 0.99, obviouly, the numerical error is (very) large.

Best
Libo

How to run this on GPU?

Just setting the device is not useful. Is there any special trick to let it run on GPU?

Implementing the 1D wave equation

Thank you for the great contribution. PINNs have been an amazing tool in scientific computing and this enhancement makes it better. I replicated the code of burger's equation and made changes to model the 1D wave equation with a source term:

@tf.function
def f_model(x,t):
    c = tf.constant(1, dtype = tf.float32)
    Amp = tf.constant(1, dtype = tf.float32)
    frequency = tf.constant(1, dtype = tf.float32)
    sigma = tf.constant(0.5, dtype = tf.float32)
    source_x_coord = tf.constant(0, dtype = tf.float32)
    
    Gaussian_impulse =  Amp * tf.exp(-(1/(sigma**2))*(x-source_x_coord)**2)
    S = Gaussian_impulse * tf.sin( 1 * tf.constant(math.pi)  * frequency * t )
    u = u_model(tf.concat([x,t], 1))
    u_x = tf.gradients(u,x)
    u_xx = tf.gradients(u_x, x)
    u_t = tf.gradients(u,t)
    u_tt = tf.gradients(u_t,t)
    f_u = u_tt + (c**2) * u_xx - S
    
    return f_u

However, the output is not quite what I expected.

I tried a couple of network architectures and learning rates, but the result seems to be the same. What would you suggest to get more accurate results?
Also, I saw how you placed the boundary conditions but I'm not sure how to specify the initial condition? Is it extracted from the exact solution?

Thanks again! Awaiting your feedback.

Is there a wrong?

mse_0_u = tf.reduce_mean(tf.square(u_weights*(u0 - u0_pred)))
mse_f_u = tf.reduce_mean(tf.square(col_weights*f_u_pred))

order your paper,

mse_0_u and mse_f_u should be:

mse_0_u = tf.reduce_mean(u_weightstf.square((u0 - u0_pred)))
mse_f_u = tf.reduce_mean(col_weightstf.square(f_u_pred))

I am not sure for this, is it right?

the problem in Helmholtz ntk

Dear @levimcclenny ,
Thanks for sharing the code, it is awesome. I have a question about helmholtz ntk, the uploaded code doesn't use ntk, could you please update the code? In addition, do you think I can use it to deal with the ultrasound data e.g. frequency = 5MHz.
Thanks a lot

How to solve Schrödinger equation with SA-PINN?

Thanks for your important contribution! I want to solve the Schrödinger equation,but i have no idea how to implement it.Could you please give some advice about it.Thank you in advance!

$$ \begin{aligned} &i h_t+0.5 h_{x x}+|h|^2 h=0, \quad x \in[-5,5], \quad t \in[0, \pi / 2], \\ &h(0, x)=2 \operatorname{sech}(x) \\ &h(t,-5)=h(t, 5) \\ &h_x(t,-5)=h_x(t, 5) \end{aligned} $$

Question about Viscous Burgers equation

 Thank you very much for sharing your code. It has been extremely helpful and inspiring. I appreciate your contributions to the development of PINN. 
 I have downloaded your source code and tried running the code related to the Viscous Burgers equation. However, I'm experiencing gradient explosions and ending up with NaN values for the loss when using the L-BFGS algorithm. Do you have any suggestions on how to address this issue?

loss increased in 1D wave equation

Dear @levimcclenny ,
Thanks for the innovative contribution. I tried to run the wave equation 1D but the loss increased all the time. I don't know why. Could you please help me check it? Thanks a lot!!