billfarrow / pcimem Goto Github PK
View Code? Open in Web Editor NEWSimple program to read & write to a pci device from userspace
License: GNU General Public License v2.0
pcimem's Introduction
== Overview ==
The pcimem application provides a simple method of reading and writing
to memory registers on a PCI card.
Usage: ./pcimem { sys file } { offset } [ type [ data ] ]
sys file: sysfs file for the pci resource to act on
offset : offset into pci memory region to act upon
type : access operation type : [b]yte, [h]alfword, [w]ord, [d]ouble-word
data : data to be written
== Platform Support ==
WARNING !! This method is platform dependent and may not work on your
particular target architecture. Refer to the PowerPC section below.
== Example ==
bash# ./pcimem /sys/devices/pci0001\:00/0001\:00\:07.0/resource0 0 w
/sys/devices/pci0001:00/0001:00:07.0/resource0 opened.
Target offset is 0x0, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x0)
PCI Memory mapped to address 0x4801f000.
Value at offset 0x0 (0x4801f000): 0xC0BE0100
== Why do this at all ? ==
When I start working on a new PCI device driver I generally go through a
discovery phase of reading and writing to certain registers on the PCI card.
Over the years I have written lots of small kernel modules to probe addresses
within the PCI memory space, constantly iterating: modify code, recompile, scp
to target, load module, unload module, dmesg.
Urk! There has to be a better way - sysfs and mmap() to the rescue.
== Sysfs ==
Let's start at with the PCI files under sysfs:
bash# ls -l /sys/devices/pci0001\:00/0001\:00\:07.0/
total 0
-rw-r--r-- 1 root root 4096 Jul 2 20:13 broken_parity_status
lrwxrwxrwx 1 root root 0 Jul 2 20:13 bus -> ../../../bus/pci
-r--r--r-- 1 root root 4096 Jul 2 20:13 class
-rw-r--r-- 1 root root 256 Jul 2 20:13 config
-r--r--r-- 1 root root 4096 Jul 2 20:13 device
-r--r--r-- 1 root root 4096 Jul 2 20:13 devspec
-rw------- 1 root root 4096 Jul 2 20:13 enable
-r--r--r-- 1 root root 4096 Jul 2 20:13 irq
-r--r--r-- 1 root root 4096 Jul 2 20:13 local_cpus
-r--r--r-- 1 root root 4096 Jul 2 20:13 modalias
-rw-r--r-- 1 root root 4096 Jul 2 20:13 msi_bus
-r--r--r-- 1 root root 4096 Jul 2 20:13 resource
-rw------- 1 root root 4096 Jul 2 20:13 resource0
-rw------- 1 root root 65536 Jul 2 20:13 resource1
-rw------- 1 root root 16777216 Jul 2 20:13 resource2
lrwxrwxrwx 1 root root 0 Jul 2 20:13 subsystem -> ../../../bus/pci
-r--r--r-- 1 root root 4096 Jul 2 20:13 subsystem_device
-r--r--r-- 1 root root 4096 Jul 2 20:13 subsystem_vendor
-rw-r--r-- 1 root root 4096 Jul 2 20:13 uevent
-r--r--r-- 1 root root 4096 Jul 2 20:13 vendor
The vendor and device files report the PCI vendor ID and device ID:
bash# cat device
0x0001
This info is also available from lspci
bash# lspci -v
0001:00:07.0 Class 0680: Unknown device bec0:0001 (rev 01)
Flags: bus master, 66MHz, medium devsel, latency 128, IRQ 31
Memory at 8d010000 (32-bit, non-prefetchable) [size=4K]
Memory at 8d000000 (32-bit, non-prefetchable) [size=64K]
Memory at 8c000000 (32-bit, non-prefetchable) [size=16M]
This PCI card makes 3 seperate regions of memory available to the host
computer. The sysfs resource0 file corresponds to the first memory region. The
PCI card lets the host computer know about these memory regions using the BAR
registers in the PCI config.
== mmap() ==
These sysfs resource can be used with mmap() to map the PCI memory into a
userspace applications memory space. The application then has a pointer to the
start of the PCI memory region and can read and write values directly. (There
is a bit more going on here with respect to memory pointers, but that is all
taken care of by the kernel).
fd = open("/sys/devices/pci0001\:00/0001\:00\:07.0/resource0", O_RDWR | O_SYNC);
ptr = mmap(0, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
printf("PCI BAR0 0x0000 = 0x%4x\n", *((unsigned short *) ptr);
== PowerPC ==
To make this work on a PowerPC architecture you also need to make a small
change to the pci core. My example is from kernel 2.6.34, and hopefully this
will be fixed for us in a later kernel version.
bash# vi arch/powerpc/kernel/pci-common.c
/* If memory, add on the PCI bridge address offset */
if (mmap_state == pci_mmap_mem) {
-#if 0 /* See comment in pci_resource_to_user() for why this is disabled */
+#if 1 /* See comment in pci_resource_to_user() for why this is disabled */
*offset += hose->pci_mem_offset;
#endif
res_bit = IORESOURCE_MEM;
/* We pass a fully fixed up address to userland for MMIO instead of
* a BAR value because X is lame and expects to be able to use that
* to pass to /dev/mem !
*
* That means that we'll have potentially 64 bits values where some
* userland apps only expect 32 (like X itself since it thinks only
* Sparc has 64 bits MMIO) but if we don't do that, we break it on
* 32 bits CHRPs :-(
*
* Hopefully, the sysfs insterface is immune to that gunk. Once X
* has been fixed (and the fix spread enough), we can re-enable the
* 2 lines below and pass down a BAR value to userland. In that case
* we'll also have to re-enable the matching code in
* __pci_mmap_make_offset().
*
* BenH.
*/
-#if 0
+#if 1
else if (rsrc->flags & IORESOURCE_MEM)
offset = hose->pci_mem_offset;
#endif
pcimem's People
Contributors
Stargazers
Watchers
Forkers
weixu8 inductiveload kanghaiyang lekensteyn shreyashnangnurwar huangxianxia idodoyo neozing faux123 wilderfield digikata sword-huang numato-viya tirfil snalli huwan wqlxx fbeneventi joaopaulosm openhacker cbtogu yncxcw nisarg01 vaibhav92 wusewolf lijie-jiang a-abraham jhana1 basseuph yangzhangwz sjokai aquantia abelardojarab ysheldon honggli sandeshghimire alticemafzzz mtp401 eliaskousk hauzlin mischief arre525 tmvector sdann hinxx w0lfzhang iomartin wade-welles portalgun-io r-novel ericurse dmarkandaya spaceradar litteng aapocketz row6 zhangrb riple zhangyanlin mistryalok singi achilikin starliiit huotui-xiaoji penguin4hire alterwood beyondsuntong rspwfpgas pouya-haghi hushunkui chenhaiwu matthewjfarrell jawsomec zaidqureshi yongliang98041092 mperov sy19870112 nanom1t qiaoxu123 giltabibian skoehler ycorrales laitianli testriscv monte-lin mng2 xinscrs neverm0 vignesh0025 xshengshe kjisu96 thschaub a1gx marcusr-xilinx 1158114251 mirfaisal-sw zjm1060 jseguisa pascal-xie udhayakumar-in8pcimem's Issues
System goes inaccessible after using ./pcimem
Hi,
I tried writing to a location using the following command.
sudo ./pcimem /sys/bus/pci/devices/0000\:0a\:00.0/resource0 0x100 w 0x1234abcd
The machine got stuck and I am not able to ping it now.
What could be the reason?
Invalid argument
Hi,
We are facing new error using pcimem that we didn't see before:
[BSC:root@15DZX23-A pcimem]$ ./pcimem /sys/devices/pci0000\:0a/0000\:0a\:00.0/resource0 0x0 b 0x44 /sys/devices/pci0000:0a/0000:0a:00.0/resource0 opened. Target offset is 0x0, page size is 4096 mmap(0, 4096, 0x3, 0x1, 3, 0x0) Error at line 111, file pcimem.c (22) [Invalid argument]
Any idea what can be the root cause for that?
A way to control request lengths?
Hey, this tool is very nice.
Curious if it can or could be enhanced to support control of the request length.
I painfully discovered that my x86 Ubuntu system was sending MemRd requests formatted as 32 bit addressing, but Length == 2... so requesting 64 bits of data.
My endpoint design (Xilinx PCIE Example Design) only supports 32 bit requests.
So my endpoint just drops the request.
I think for now I am going to have to write my own PCIe User Logic.
Fedora 33 mmap failing
on Fedora 33
lspci
b3:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
Subsystem: Mellanox Technologies ConnectX-4 Lx Stand-up dual-port 10GbE MCX4121A-XCAT
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 37
NUMA node: 0
Region 0: Memory at 3800fe000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at fbe00000 [disabled] [size=1M]
Capabilities:
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core
ls /sys/bus/pci/devices/0000:b3:00.0/ -l
total 0
-r--r--r--. 1 root root 4096 Mar 1 11:41 aer_dev_correctable
-r--r--r--. 1 root root 4096 Mar 1 11:41 aer_dev_fatal
-r--r--r--. 1 root root 4096 Mar 1 11:41 aer_dev_nonfatal
-r--r--r--. 1 root root 4096 Feb 21 09:25 ari_enabled
-rw-r--r--. 1 root root 4096 Mar 1 11:41 broken_parity_status
-r--r--r--. 1 root root 4096 Feb 21 09:25 class
-rw-r--r--. 1 root root 4096 Feb 21 09:25 config
-r--r--r--. 1 root root 4096 Mar 1 11:41 consistent_dma_mask_bits
-r--r--r--. 1 root root 4096 Mar 1 11:41 current_link_speed
-r--r--r--. 1 root root 4096 Mar 1 11:41 current_link_width
-rw-r--r--. 1 root root 4096 Mar 1 11:41 d3cold_allowed
-r--r--r--. 1 root root 4096 Feb 21 09:25 device
-r--r--r--. 1 root root 4096 Mar 1 11:41 dma_mask_bits
lrwxrwxrwx. 1 root root 0 Feb 21 09:25 driver -> ../../../../bus/pci/drivers/mlx5_core
-rw-r--r--. 1 root root 4096 Mar 1 11:41 driver_override
-rw-r--r--. 1 root root 4096 Mar 1 11:41 enable
lrwxrwxrwx. 1 root root 0 Mar 1 11:41 firmware_node -> ../../../LNXSYSTM:00/LNXSYBUS:00/PNP0A08:03/device:b6/device:b9
drwxr-xr-x. 16 root root 0 Feb 21 09:25 infiniband
drwxr-xr-x. 4 root root 0 Feb 21 09:25 infiniband_mad
drwxr-xr-x. 3 root root 0 Feb 21 09:25 infiniband_verbs
-r--r--r--. 1 root root 4096 Mar 1 10:13 irq
drwxr-xr-x. 2 root root 0 Feb 24 14:45 link
-r--r--r--. 1 root root 4096 Mar 1 11:41 local_cpulist
-r--r--r--. 1 root root 4096 Mar 1 11:41 local_cpus
-r--r--r--. 1 root root 4096 Mar 1 11:41 max_link_speed
-r--r--r--. 1 root root 4096 Mar 1 11:41 max_link_width
-r--r--r--. 1 root root 4096 Mar 1 10:13 modalias
-rw-r--r--. 1 root root 4096 Mar 1 11:41 msi_bus
drwxr-xr-x. 2 root root 0 Feb 24 14:45 msi_irqs
drwxr-xr-x. 3 root root 0 Feb 21 09:25 net
-rw-r--r--. 1 root root 4096 Feb 21 09:25 numa_node
-r--r--r--. 1 root root 4096 Mar 1 11:41 pools
drwxr-xr-x. 2 root root 0 Feb 24 14:45 power
drwxr-xr-x. 3 root root 0 Feb 21 09:25 ptp
--w--w----. 1 root root 4096 Mar 1 11:41 remove
--w-------. 1 root root 4096 Mar 1 11:41 rescan
--w-------. 1 root root 4096 Mar 1 11:41 reset
-r--r--r--. 1 root root 4096 Feb 21 09:25 resource
-rw-------. 1 root root 33554432 Mar 1 11:41 resource0
-rw-------. 1 root root 33554432 Mar 1 11:41 resource0_wc
-r--r--r--. 1 root root 4096 Feb 21 09:25 revision
-rw-------. 1 root root 1048576 Feb 21 09:25 rom
-rw-r--r--. 1 root root 4096 Mar 1 11:41 sriov_drivers_autoprobe
-rw-r--r--. 1 root root 4096 Feb 21 09:25 sriov_numvfs
-r--r--r--. 1 root root 4096 Mar 1 11:41 sriov_offset
-r--r--r--. 1 root root 4096 Mar 1 11:41 sriov_stride
-r--r--r--. 1 root root 4096 Feb 21 09:25 sriov_totalvfs
-r--r--r--. 1 root root 4096 Mar 1 11:41 sriov_vf_device
lrwxrwxrwx. 1 root root 0 Feb 21 09:25 subsystem -> ../../../../bus/pci
-r--r--r--. 1 root root 4096 Feb 21 09:25 subsystem_device
-r--r--r--. 1 root root 4096 Feb 21 09:25 subsystem_vendor
-rw-r--r--. 1 root root 4096 Feb 21 09:25 uevent
-r--r--r--. 1 root root 4096 Feb 21 09:25 vendor
-rw-------. 1 root root 0 Mar 1 11:41 vpd
program output
sudo ./pcimem /sys/bus/pci/devices/0000:b3:00.0/resource0_wc 0 w
/sys/bus/pci/devices/0000:b3:00.0/resource0_wc opened.
Target offset is 0x0, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x0)
Error at line 111, file pcimem.c (22) [Invalid argument]
Any ideas?
[158333.439198] x86/PAT: pcimem:922560 conflicting memory types ca000000-ca800000 uncached-minus<->write-combining
162676.233287] x86/PAT: pcimem:978227 conflicting memory types ca000000-ca800000 uncached-minus<->write-combining
[162676.233289] x86/PAT: memtype_reserve failed [mem 0xca000000-0xca7fffff], track uncached-minus, req uncached-minus
Trying to access BAR0 of a Mellanox ConnectX-5.
Works fine on RHEL 7.7.
On RHEL it appears the BAR is mapped much higher in memory (perhaps the BIOS has the above 4G flag set)?
First time running the same logic on Ubuntu.
Running on Ubuntu produces the error.
more /sys/devices/pci0000:00/0000:00:1b.4/0000:02:00.0/resource
0x00000000ca000000 0x00000000cbffffff 0x000000000014220c
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x00000000f7100000 0x00000000f71fffff 0x0000000000046200
0x00000000ce000000 0x00000000ceffffff 0x000000000014220c
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
02:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
Subsystem: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at ca000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at f7100000 [disabled] [size=1M]
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
Capabilities: [c0] Vendor Specific Information: Len=18
Capabilities: [40] Power Management version 3
Capabilities: [100] Advanced Error Reporting
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1c0] Secondary PCI Express
Capabilities: [230] Access Control Services
Capabilities: [320] Lane Margining at the Receiver
Capabilities: [370] Physical Layer 16.0 GT/s
Capabilities: [420] Data Link Feature
Kernel driver in use: mlx5_core
Extend use cases to /dev/mem
I have issues using dd to access PCI memory via /dev/mem. This tool is a great replacement:
./pcimem /dev/mem/ ...
Docs could be updated to reflect this.
Info : Maximum size of pcimem we can access
Equation : (BAR_SIZE - page_size) - 1
e.g. In my Case,
Page size = 4096
BAR_SIZE = 1 MB
so, (1048576 - 4096 ) - 1 = 1044479
[Not an 'issue'] Doubt regarding PCI resource size
Hi,
Is there any restriction on setting the resource file size (I do it through Vivado).
When I change the size from 128MB to 256MB, lspci -vv stops displaying that resource corresponding to that.
Question posted in Stackoverflow
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.